# Learned FH% regression: design and collection plan Context: `g_uJointFHThousandths`/`g_uBetaFHThousandths` (`data.c`) are 5D/4D histogram tables predicting "will this move/node fail high," used to gate LMR/LMP admission (alpha side) and static null-move cutoff + node-type refinement (beta side). This doc designs their replacement: two small logistic regressions, one per side of the window, fit offline and baked into the engine as quantized integer weight vectors. ## Why move off the table 1. **Dimension growth is combinatorial, sample growth isn't.** The shipped alpha table is already `12x11x11x17x11` ~= 268K cells; the NNUE collection run (`../dual_table_run_nnue/README.md`) found 121,192 of 271,524 cells still below the 500-sample floor even after a multi-day, 16-shard run. Adding even one more binary axis roughly doubles the cell count and thins every existing cell further. A regression's cost per new feature is one more weight, not a multiplicative storage/sample penalty. 2. **New discriminating signals don't fit cleanly as new axes.** The `docs/DEEPER_PRUNING.md` HASHCAP_COLLECTION side-study found two real effects -- node-type (CUT vs ALL/PV) carries a measured ~9x effect on leftover-move FH%, and whether the hash move was a capture carries a separate ~2x effect, and the two *interact* (a CUT-node-survived-this-far signal is itself evidence the CUT guess was wrong). Wiring either into the existing table means a full axis-dimension expansion; a regression just adds features and an interaction term. 3. **Measured discrimination is real and worth capturing properly.** Pulled directly from `../dual_table_run/pickles/baked_now.pkl` (`compare_improve_axis.py` plus one-off marginal scripts written this session): Alpha side (`iEval <= iAlpha`), global marginal rate ratio per axis: | axis | ratio (max/min FH%) | |---|---| | gap | 1647x | | improve | 128x | | dumb (hang estimate) | 96x | | late (move lateness) | 40x | | depth | 24x | Beta side (`iEval >= iBeta`), using **miss-rate** (`100% - FH%`) ratio, since raw FH% sits near a 95-99% ceiling where a percentage-ratio understates real discrimination: | axis | ratio (miss-rate max/min) | |---|---| | depth | 155x | | gap | 91x | | improve | 12x | | own_improve | 4.8x | Note depth is the *strongest* axis on the beta side and a middling one on the alpha side -- a real, measured asymmetry, not an assumption. This is the direct evidence behind building **two independent regressions** rather than one shared model: even a feature both sides have in common carries a different weight in each regime. ## Model family: logistic regression, not trees or a net Considered and rejected for the inner loop: - **Small gradient-boosted tree ensemble** (e.g. 50 trees, depth 4): finds interactions automatically, but each split is a data-dependent branch the predictor can't learn a pattern for -- ~15-20 cycles mispredict penalty *per split*, 50 trees x 4 levels is in the ~1us range. Disqualifying at the throughput this needs to run at (every candidate move, every node). - **Small NNUE-style net**: fast once SIMD'd, but heavier infra for no clear win over logistic regression at this feature count, and NNUE eval itself is already the dominant per-node cost in this engine (~400-500K nps vs HCE's ~1.3M+) -- not the place to add more float-heavy inference. - **Logistic regression**: one dot product (~16-20 terms) + one sigmoid. Fully branch-free, pipelines well, same cost class as the existing Wilson-bound table lookup + `MIN`. This is the one we're building. **Quantization**: fit in float offline (scikit-learn `LogisticRegression`, seconds on commodity CPU for millions of rows) exactly like `tune_eval_dna.py`'s DNA constants are tuned in float and only baked to engine-native representation afterward. Quantize the learned float weights to fixed-point integers (choose the scale *after* seeing real weight magnitudes -- don't squeeze to int16 preemptively, there's no storage pressure here forcing it the way there was with table axes). Inference: `score = (sum(w_i * f_i)) >> shift`, int64 accumulator, then one lookup into a small precomputed sigmoid table keyed on a coarse bucket of `score` -- no floating point anywhere in the hot path, no runtime `exp()`/`log()`. ## Nonlinear raw quantities need an explicit feature transform Logistic regression is linear in log-odds *of the features you feed it*, not of whatever raw quantity you happened to measure. If the true relationship between depth and log-odds(FH) is logarithmic (which is why Stockfish's own reduction formula is `log(depth)*log(moveIndex)` to begin with), feeding raw depth in forces the model to approximate a curve with a straight line -- transform first (`log(depth)`, `log(moveIndex)`), don't rely on the model family to discover the right shape. Checked directly against the alpha table's real data (`baked_now.pkl`, marginalized per axis, logit = `log(p/(1-p))`): - **gap**: logit deltas per bucket are `-1.88, -2.30, -0.81, -0.48, -0.50, -0.04, -0.07, -0.23, -0.28, -0.85` -- large early drops that flatten hard, classic compressive-transform shape. `log(gap)` (or `log(gap + const)`) is the right starting point for a continuous feature; note the *existing* axis's bucket boundaries already encode some of this nonlinearity implicitly via non-uniform bucket widths, so this isn't a new discovery so much as confirmation to carry forward explicitly. - **depth**: trends down through bucket ~12 but not cleanly log-shaped (deltas bounce: `-0.12, -0.27, -0.43, -0.12, -0.06, -0.04, -0.31, ...`), then goes noisy/non-monotonic past depth 13 (`+0.79`, `-0.90` swings) -- consistent with the NNUE README's own finding that per-cell sample counts collapse toward zero past depth 13. Use `log(depth)` as the default transform (matches every other engine's own formula and the early-bucket trend direction), but don't trust the depth>13 region's shape from this data -- validate against the regression's own residuals once real feature-level (not axis-marginal) data exists, and treat depth>13 behavior as an extrapolation, same caution the existing table already applies there. - **move ordinal**: the existing `late` axis (alpha side) is the closest historical proxy and gives a textbook-clean compressive shape -- logit deltas shrink smoothly from `-0.465` down to near-zero by bucket ~27 (`-0.013`), then drift slightly positive in the extreme tail (bucket 32, a wide catch-all bucket, likely a different population mix rather than a reversal of the real trend). This is about as strong a confirmation of `log(ordinal)` as old data can give. Combined with the fact that Crafty and Stockfish both use a log-of-move-index term regardless of which side of the window they're on, `log(ordinal)` is adopted for **both** the alpha and beta side's move-ordinal feature, not left as a side-specific guess. - **improve** (alpha side, axis 4): trends up with bucket index but without gap's clean compressive shape -- deltas are noisy (`+0.85, +0.10, +0.46, +0.63, +1.51, +0.74, -0.05, +0.14, +0.51, -0.20`) though the overall direction holds across 11 buckets backed by billions of tries each. Likely doesn't need an *additional* transform beyond whatever the existing bucket boundaries already encode -- try the raw underlying continuous quantity untransformed first, confirm against the smoke test's own fresh data. - **improve / own_improve (beta side, axes 2-3)**: noisier still. `own_improve` in particular shows a sharp level-shift -- bucket 0 sits at 98.48%, every other bucket (1-10) sits in a tighter 92-96% band with alternating-sign deltas -- which smells more like "own_improve == 0" mattering as a distinct state than like a smooth magnitude effect. Worth testing an explicit `(own_improve == 0)` indicator plus a separate continuous magnitude feature against a single continuous encoding once real data exists, rather than assuming either shape now. - **dumb (hang estimate, alpha side, axis 1)**: non-monotonic, no smooth shape at all -- a huge spike at bucket 10 riding on a tiny 8.8M-try sample (vs. neighboring buckets' billions), bucket 1 also spiking well above its neighbors. Reads as a genuinely **categorical** signal (discrete hang-severity tiers), not a continuous quantity -- which confirms the decision already made above (the new, redesigned "move value" feature is a continuous signed SEE/hang-estimate, not a transform of this old discrete axis) rather than suggesting a log transform was missing here. General rule for this project: any raw feature suspected non-linear in log-odds gets checked against real axis-marginal data (same style as above) before locking its transform, not assumed from other engines' formulas alone. Where no historical axis exists to check against (every brand-new feature: `is_whole_node`, the running counter, FH%-as-feature, both cross terms, and the *new* continuous move-value definition), the smoke test's own output is the first real opportunity to check -- see "Validating against the existing table" below for how that interacts with trusting the smoke test's small sample size. ## One model, not two: NNUE-only collection (decided 2026-10-24) Originally planned as two independent collection runs (HCE and NNUE), matching `dual_table_run`/`dual_table_run_nnue`'s own split. Revised: **collect once, against NNUE only, and let HCE use the same model.** Rationale: NNUE is the strategic direction going forward, and running two full multi-day collections for a model that's meant to retire anyway isn't worth the cost. This is a reasonable call but carries a named, specific risk worth remembering rather than assuming away: the earlier HCE-vs-NNUE table comparison (see `dual_table_run_nnue/README.md`) found the **alpha** table's predictions tracked closely between the two evals (-0.11pp mean delta, flat across depth) -- clean support for reuse there -- but the **beta** table showed a real, systematic divergence (NNUE -0.6pp overall, growing to -1.1 to -1.9pp from depth bucket 2 on), plus a separate, never-investigated 3-9pp gap in one narrow alpha-side slice (depth=1, dumb=0, gap=0, early leftover moves). Before trusting one NNUE-trained model under an HCE build, the "rigorous spot-check against high-confidence table cells" step below should be run against **both** the HCE and NNUE tables' cells, not just whichever eval the collection happened to run under -- if the beta-side predictions land close to HCE's own historical cells too, that confirms the simplification; if not, that's exactly the thing worth knowing before shipping one model for both evals. Practically this requires no change to the collection design below -- `FH_COLLECT_RECORD`/the hook API never reference `g_fUseNnueEval` at all, so the only real implications are: the collection run itself uses `typhoon_nnue` exclusively, and the eventual runtime dispatch in `searchsup.c` collapses to "always use the one model," no `g_fUseNnueEval` branch needed. ## Scope: MWS (null-window) nodes only, for now Both regressions are trained and used only where `iAlpha + 1 == iBeta` (a null-window/MWS child). At a true PV node (`alpha < eval < beta` is a real, non-degenerate state), "fail high" isn't a well-defined single question the way it is here -- collecting and modeling PV nodes is explicitly **deferred as future work**, not attempted in this pass. The alpha/beta branch below is exhaustive specifically because MWS makes `iEval` always either `<= iAlpha` or `>= iBeta`, never strictly between: ```c if (iEval <= iAlpha) { uPredictedFHThousandths = RegressionA(...); } else if (iEval >= iBeta) { uPredictedFHThousandths = RegressionB(...); } ``` ## Shared design rules (both regressions) 1. **`is_whole_node` sentinel bit, plus a dedicated `no-move` tier category** -- not a raw-zero convention. The NNUE collection run's own bucket-0/1/2 collision bug (raw `late` values 0, 1, and 2 all landing in shipped bucket 0 because the placeholder and a real low-ordinal population's natural value both mapped to zero) is exactly the failure mode an implicit zero-as-"absent" convention invites. An explicit indicator bit plus a genuinely separate categorical value avoids repeating it. 2. **`moves_known` bit + mean-imputation for total legal move count.** In production, moves aren't generated until after the null-move and hash- move tries, so the real count is genuinely unknown for the whole-node, null-move, and hash-move predictions -- not zero, unknown. Pair the feature with a binary indicator and impute the dataset mean when unknown; the collection *build* can eagerly generate moves for every logged event regardless of production's lazy order (collection already pays extra cost for ground-truth correctness -- see LMP-disabling precedent below), so training data always has the real count even where production inference won't. 3. **Node-type guess fed in must be the raw, parent-derived guess only** -- never the *refined* guess this same regression produces. Feeding a model's own (even indirectly derived) output back in as an input is leakage: it'll look great in training and be meaningless at inference time, where the refined guess doesn't exist yet. 4. **`in_check` is dropped entirely.** It's a hard upstream gate -- no pruning, reduction, or null-move is ever attempted while in check -- so the training population never contains an `in_check=true` row by construction. A feature with zero variance in its own training population carries zero information; it's a precondition for the model running at all, not something for the model to weigh. 5. **Evidence is the move's own calibrated FH% (Wilson-bound estimate), not a raw history/continuation counter.** A raw counter has no natural zero point or scale comparable to a percentage; the move's own FH% estimate is already computed, already calibrated, and lives on the same 0-100 scale as every other probability-flavored feature. 6. **Generalized "good moves tried without a cutoff" running counter.** Replaces the narrower "was the hash move a capture" binary flag with a loop-state feature: the count (and fraction of moves-so-far) of winning/even-capture moves already tried at this node that failed to produce the relevant outcome. Mutates as the move loop progresses (zero before any move, incrementing on each surviving good move) -- directly operationalizes the "a CUT guess that survives this long is probably wrong" finding rather than encoding one static fact about the hash move alone. 7. **Explicit hand-built cross terms for measured interactions.** Plain logistic regression won't discover an interaction on its own -- only a tree ensemble does that automatically, and we ruled trees out on speed. Where there's real measured evidence of an interaction (node-type x running-counter, per the ~9x CUT effect), add the product as its own feature explicitly. 8. **Nonlinear raw quantities get transformed before entering the model** -- see above. ## Feature list -- alpha side (`iEval <= iAlpha`) Predicts: did this move's (or the whole node's eventual best) resulting score end up `>= iBeta`. **Node-level** (computed once per node; shared by the whole-node row and every move-level row at this node): | # | feature | type / transform | |---|---|---| | 1 | `gap` = iAlpha - iEval | continuous, `log(gap)` | | 2 | remaining depth | continuous, `log(depth)` | | 3 | node-type guess (CUT/ALL/PV), **parent-derived only** | categorical | | 4 | improving | binary/continuous + trend (1st derivative) | | 5 | enemy improving | binary/continuous + trend | | 6 | hang/dumb estimate (position-wide) | continuous | | 7 | total legal move count | continuous, paired with `moves_known` bit + mean-imputation | | 8 | phase/material bucket | categorical, coarse | | 9 | `is_whole_node` | binary sentinel | **Move-level / loop-state** (one row per move actually tried: `null`, hash, good-capture, killer, countermove-match, leftover-quiet, losing-capture -- plus the `no-move` sentinel row paired with `is_whole_node=1`): | # | feature | type / transform | |---|---|---| | 10 | move tier tag (incl. `null`, `no-move`) | categorical | | 11 | move value, signed (SEE/IID/hang-estimate; +300/0/-900 style) | continuous | | 12 | move ordinal, raw | continuous, `log(ordinal)` | | 13 | move ordinal, normalized (ordinal / total legal moves) | continuous | | 14 | move's own FH% (Wilson-bound estimate) | continuous, 0-100 scale | | 15 | running counter: good/even captures tried without a cutoff, raw + fraction | continuous | | 16 | node-type-CUT x running-counter (cross term) | continuous | ## Feature list -- beta side (`iEval >= iBeta`) Predicts: did this move's (or the null-move probe's, or the whole node's eventual best) resulting score end up `>= iBeta`. Same label definition as alpha side. **Node-level:** | # | feature | type / transform | |---|---|---| | 1 | `gap` = iEval - iBeta | continuous, transform TBD -- check logit-linearity against this side's own data before assuming `log(gap)` carries over unchanged from the alpha side | | 2 | remaining depth | continuous, `log(depth)` -- strongest single axis measured on this side (155x miss-rate ratio) | | 3 | node-type guess, **parent-derived only** | categorical | | 4 | improving / own-improving | binary/continuous + trend, both already real axes in the shipped table | | 5 | total legal move count | continuous, `moves_known` bit + mean-imputation | | 6 | phase/material bucket | categorical | | 7 | `is_whole_node` | binary sentinel | **Move-level / loop-state** (`null`, hash, good-capture, killer, countermove-match, leftover-quiet, losing-capture, `no-move`): | # | feature | type / transform | |---|---|---| | 8 | move tier tag | categorical | | 9 | move value, signed | continuous -- **flagged below, not yet trusted** | | 10 | move ordinal, raw + normalized | continuous, `log(ordinal)` for raw | | 11 | move's own FH% (Wilson-bound estimate) | continuous, 0-100 | | 12 | running counter: good/even captures tried without preserving the cutoff, raw + fraction | continuous | | 13 | node-type-CUT x running-counter | continuous | ## Open items -- measure before trusting, don't assume symmetry with alpha - **Move value / SEE on the beta side is reported to have measured weak in an earlier (undocumented) attempt, and the reason isn't understood.** Candidate explanations, none verified: (a) a ceiling effect -- beta-side outcomes already sit at 92-99%, leaving little residual probability for any feature to explain; (b) collinearity with `gap` -- the node's static eval may already reflect whatever material swing produced the advantage, making a candidate move's own SEE partly redundant once gap is in the model; (c) the original measurement itself may have been flawed (bad bucketing, a confound) rather than the signal being genuinely weak. Needs its own small, targeted diagnostic (same shape as `HASHCAP_COLLECTION`) logging move value alongside outcome on this population specifically, before deciding to keep, drop, or redefine it. - **The alpha-side `dumb`/hang-estimate axis measured strong (96x); its beta-side analog is not assumed to be the same feature.** "Is material propping up this favorable eval actually under threat" is a different question than "is material hanging in general" -- needs its own definition and its own measurement, not a copy-paste from the alpha side. - **Speculative features to log now (cheap to add to the collection build) and decide on later via measurement, not logged with any strong prior they'll survive:** - static eval magnitude itself, not just `gap` (a `+900`-eval node and a `+50`-eval node with the same gap may not behave identically) - ply-from-root, distinct from remaining depth (horizon/history-evidence maturity effects near the root) - repetition/50-move-adjacency (probably weak, cheap to check and drop) ## Collection **Format: dense binary records, not text logs.** The existing `FH_TABLE_COLLECTION` instrumentation writes full per-node trace text via `Trace()`, which produced 13GB across 16 shards for the NNUE run and, per observation on this session's `sd 12` shallow-tier shard, grows especially fast on the shallow/high-diversity tier (`shard_15.log.bz2`, 5.3GB *compressed*). Fix this structurally rather than stripping after the fact: have the collection build emit one small fixed-width struct per logged event (whole-node row, or one row per move/null-move tried) directly to a per-shard binary file -- feature values and the outcome label, no text formatting, no field names repeated per line. This is both far smaller on disk and faster to load for training (straight into numpy via a dtype, no text parsing). **Keep the deep/shallow/medium shard tiering** (same idea as the existing run: one shard `sd 20`, one `sd 12`, the rest `sd 16`) -- the shallow tier's high position-diversity is valuable on its own terms and isn't the thing causing the disk problem; the disk problem was the text-log format, which the binary-record change solves independently of tiering. **Collection-build changes needed:** 1. Disable LMP during alpha-side collection (existing `FH_TABLE_COLLECTION` precedent -- still needed, same reasoning). 2. Disable the static reverse-futility shortcut during beta-side collection (new requirement, same reasoning as #1: the shortcut would otherwise short-circuit the whole node before any real moves -- or even the whole-node row itself -- get a chance to be logged). 3. Eagerly generate moves for every eligible node regardless of whether null-move/hash-move would have short-circuited it in production, so `total legal move count` is real (not imputed) in every training row. 4. Log one `is_whole_node` row per eligible node before the move loop starts, then one row per move tried thereafter (`null` first where applicable, then hash, then the rest in generation order) -- all gated on the same `iEval <= iAlpha` / `iEval >= iBeta`, MWS-only eligibility check used today. ## Validating against the existing table: sanity check vs. rigorous gate The existing table's best-populated cells carry enormous sample sizes -- the alpha table alone has individual `(depth,dumb,gap,late)` cells with over 2 billion tries (see the `improve`-axis within-cell check earlier in this doc's history). That's real, cheap ground truth, and it should be used twice, at two different points, for two different purposes -- conflating them risks either dismissing a good model too early or trusting a bad one too late. 1. **Cheap sanity check, against the smoke-test model.** Pick 3-4 of the table's highest-tries cells, build the matching feature vector, and confirm the smoke model's predicted FH% is at least in the right ballpark (right order of magnitude, right direction) -- not a rigorous statistical comparison, just a bug catcher. The smoke test's training sample is deliberately small, so a real mismatch here is more likely to mean a sign error, a scaling mistake, or a transform bug than a genuine quality problem -- treat a failure here as "go find the bug," not "the approach doesn't work." Run this right after the smoke test's model is fit, before investing in the full run. 2. **Rigorous pre-integration gate, against the full-run model.** After the real multi-day collection and fit, compare the final model's predictions against the same high-tries cells as the primary calibration check before any production code changes -- this is the one that actually answers "is this model trustworthy," since both sides of the comparison now have enough data to be statistically meaningful. Report predicted-vs-actual across a spread of cells (not just the handful used for the quick sanity pass), the same binned-calibration discipline the table itself would need if its own boundaries were ever re-derived. ## Plan 1. ~~Finalize this doc.~~ 2. ~~Instrument the collection build.~~ Landed 2026-10-24: `FH_REGRESSION_COLLECTION` (replaces `FH_TABLE_COLLECTION` outright -- see `GNUmakefile`) in `chess.h`/`search.c`/`searchsup.c`/`fhcollect.c` (new file)/`main.c`/`root.c`. Call-site design matches the "function calls, not big `#ifdef`s in the middle of `Search()`" guidance: three always-callable hooks (`FhCollectNodeEntry`/`FhCollectMoveTried`/ `FhCollectNodeExit`, no-op inline stubs in a non-collection build) plus `FhClassifyMoveTier` (decodes `generate.c`'s packed `_ScoreAllMoves` sort key -- hash/good-capture/killer/countermove/leftover-quiet/ losing-capture -- back into a tier, checked in the same priority order that scoring function documents, since `COUNTERMOVE_TIER_BONUS` shares a bit with `GOOD_MOVE` and needs an exact match, not a bare AND). A `FH_COLLECT_PENDING` struct, declared as an ordinary local in `Search()` (deliberately not stored in `ctx`/`PLY_INFO` -- recursion would clobber a shared copy before the parent node's own `NodeExit` ran), carries state from node entry through each move tried to the final whole-node commit. Per-thread binary files (`fhcollect_threadNN.bin`, one `FH_COLLECT_RECORD` per logged event, buffered in `FH_COLLECT_BUFFER_RECORDS`-sized batches) replace the old histogram entirely -- no accumulation/merge code needed anywhere, so the corresponding `root.c`/`script.c` copy-out-and-dump blocks were deleted outright, not just renamed. Verified end-to-end on a smoke run (see below) before moving on. 3. **Smoke test before the real run**: ran a single-shard, `sd 6`, one-position smoke test (`--cpus 1`, LMP confirmed disabled via "Overt move skips: LMP=0 moves" in the engine's own output). Both build profiles (`FH_REGRESSION_COLLECTION=1 PERF_COUNTERS=1 MP=1` and the normal `GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1` production profile) compile clean with no new warnings. Parsed the resulting 16,384-record binary file in Python: zero suspicious rows, sane table(alpha/beta)/tier/label distributions (dominated by `LEFTOVER_QUIET` as expected with LMP off, a plausible handful of `NULL`/`HASH`/`KILLER`/rare `COUNTERMOVE` rows), and matching `NULL`/`NO_MOVE` row pairs sharing identical position signature/gap/ depth at the same node, confirming the stash-then-commit two-phase design works correctly across the node's lifetime. **Full soup-to-nuts pass completed 2026-10-24**, including a real `sklearn` fit and a spot-check against the existing historical table -- see "Running, training, and validating" below for the commands and `regression_run/` for the scripts. Four real bugs were found and fixed along the way, all caught by this smoke-testing, none by code review alone: 1. `FhClassifyMoveTier`'s capture check used `TRUE == IS_CAPTURE_OR_PROMOTION(mv)` -- that macro is a bitmask test (`mv.uMove & 0x0FF00000`), not a 0/1 boolean, so it was almost always false for real captures. Every capture silently misclassified as a quiet move (mostly `LEFTOVER_QUIET`, some `KILLER` -- a losing capture's negative `iValue` sign-extends with its high bits all set, coincidentally matching the killer- bit test). Fixed to a plain nonzero check, matching every other real call site in the codebase. 2. The per-move `move_value` feature for quiet moves was reading `g_iPSQT[mv.pMoved][mv.cTo]` (a positional square-preference number) via the packed sort key, not a material-safety estimate -- a different signal than the one actually designed and validated (the old table's `dumb` axis, 96x discrimination on the alpha side). Fixed to call `EstimateHangingMaterialCp` explicitly for quiet moves, matching `searchsup.c`'s own existing call to it. 3. `GetMoveFailHighWilsonUpperBoundThousandths` returns a confidently-low value for a move identity with zero evidence (the table's own `n=1,k=0` cold-start convention), not a neutral "unknown" -- indistinguishable from a genuinely well-evidenced safe move without more context. Added `uMoveFhAttempts` (`GetMoveFailHighAttempts`, new function) as a companion raw evidence-count field. 4. `total_non_pawn_material` (the phase/material feature, itself missing from the first implementation -- see the feature audit below) initially summed `pos->uNonPawnMaterial[color]` directly, which includes each side's own king at `VALUE_KING` == `INFINITY_SCORE` (32767) -- a ~65534 constant swamping the real ~0-6725 of actual piece material riding on top of it. Fixed by subtracting `VALUE_KING` per side, the same convention `eval.c`'s own `uArmyScaler` already uses. **Feature audit against this doc's own tables** (done after the above fixes, before trusting anything further): every listed feature is present and correctly wired except alpha-side node-level feature #6, `hang/dumb estimate (position-wide)` -- never implemented at all. Decision: leave it as a documented placeholder (0) rather than build new code for it now -- the old table *also* pinned this exact whole-node slot to a hardcoded 0 (there's no natural move-independent analog of `EstimateHangingMaterialCp` without a more expensive full-board threat scan), so this isn't a regression, just an unfilled slot with real historical precedent. Revisit only if the trained model's calibration specifically implicates missing whole-node material-safety signal. The MP/`StartParallelSearch` split path remains deliberately uninstrumented (collection runs at `--cpus 1`, so this path is never exercised in practice). 4. Full multi-shard, multi-day collection run (binary format, same deep/shallow/medium tiering), against `typhoon_nnue` only (see "One model, not two" above). 5. Fit both regressions offline, quantize, validate calibration (predicted vs. actual FH%, binned, same discipline as validating the existing table) before touching production code. 6. Integrate into `search.c`/`searchsup.c`: replace the table-lookup call sites with the quantized dot-product + sigmoid-lookup inference, gated the same way the existing HCE/NNUE table pair is selected today (`g_fUseNnueEval && g_fNnueWeightsLoaded`-style runtime switch -- TBD whether this model needs a similar HCE/NNUE split or can be eval- agnostic; check empirically rather than assume). 7. Full validation ladder per `docs/DEEPER_PRUNING.md`'s existing discipline: wiring sanity (`DEBUG=1 TEST=1` + `debug_smoke_test.sh`), per-move FH-rate instrumentation, `sd 10` and `sn 5000000` suite diffs on the three curated suites, `match_play.py` gated self-play (`LOWER95 >= 0.5`) before this ever ships as the default. ## Running, training, and validating Everything below lives in `regression_run/` (sibling of `src/`, same convention as `dual_table_run/`), and all of it already works end to end against real smoke-test data -- this is the actual, exercised procedure, not a forward-looking plan. ### 1. Build the collection binary ```sh cd src gmake clean # after any chess.h change touching many TUs gmake -j5 PERF_COUNTERS=1 MP=1 FH_REGRESSION_COLLECTION=1 ``` Produces `src/typhoon` with the collection hooks live. **Never ship this binary for anything other than a collection run** -- LMP and the beta- side static-null-move shortcut are both disabled under this flag (see "Collection-build changes needed" above), a real strength regression in normal play. ### 2. Launch a collection run Same invocation convention as the rest of this project's ECM testing (CLAUDE.md's "Testing: the ECM tactical suite" section) -- `--cpus 1`, book disabled, `easy` to stop background pondering from contaminating the logs, `sd ` not `st ` for a reproducible, load- independent run: ```sh mkdir -p /tmp/typhoon/ && cd /tmp/typhoon/ /path/to/typhoon --cpus 1 --hash 256m \ --logfile /tmp/typhoon//run.log \ --batch --command "force; book name /nonexistent.book.bin; easy; sd 10; \ script /usr/home/scott/typhoon/tests/.ep_" ``` Produces `fhcollect_thread00.bin` (and `fhcollect_threadNN.bin` per helper thread, if `MP=1` and `--cpus > 1`) in the current directory -- one `FH_COLLECT_RECORD` per logged event, 64 bytes each as of this writing (confirmed via `offsetof`/`sizeof` against the real compiled struct, not assumed -- see `regression_run/load_records.py`'s own docstring for why that matters). A 90-position suite at `sd 10` runs in well under a minute and produces on the order of tens of millions of records to low hundreds of GB for the real multi-shard run -- launch in the background (`nohup ... &` or `run_in_background`), don't block the foreground on it. ### 2b. Or launch a real multi-shard run (16 shards, like dual_table_run*) `regression_run/build_shards.py` is `dual_table_run_nnue/build_shards.py` copied verbatim -- it's pure EPD-pool slicing, no awareness of which table format reads the shard files it builds, so it needed no changes: ```sh cd regression_run python3 build_shards.py --repo-root .. --num-shards 16 \ --medium-positions-per-shard --out-dir shards ``` Then `launch_shards.sh ` -- same deep/shallow/medium depth tiering convention as the dual-table runs (1 `sd20`, 1 `sd12`, 14 `sd16`, overridable via env vars, see the script's own header comment for the all-`sd10` smoke-test invocation), using `../src/typhoon_nnue` (NNUE-only collection, per the decision above): ```sh ./launch_shards.sh my_run_name ``` **One real difference from the old `FH_TABLE_COLLECTION` shard scripts, worth understanding before relying on this**: there, each shard's *data* lived in its own `--logfile`, so all 16 could safely share one working directory. Here, the data is `fhcollect_thread00.bin` -- named only by **thread number within a process** (`chess.h`'s `FH_COLLECT_WRITER` comment), since the file- naming scheme was designed around multiple *threads* in one engine (`MP=1`), not multiple independent shard *processes*. 16 shards sharing a directory would all collide on the same filename. Fixed with no C changes: `launch_shards.sh` gives each shard its own working directory (`runs//shard_NN/`) before `exec`ing the engine, so each one's `fhcollect_thread00.bin` is a distinct file on disk. **Recombining shards is simpler than the old design, not harder**: `load_records.py`'s `glob`-and-concatenate already treats "every matching file under a directory" as one dataset -- pointing it at `runs//shard_*/fhcollect_thread00.bin` (via the `pattern` argument) *is* the recombination step. There's no `aggregate.py`/ `bake.py`-style merge needed, because there's no in-process histogram to sum across shards in the first place -- concatenating raw rows is the whole operation. **Checking progress and coverage while shards are still running**: ```sh ./dashboard.py my_run_name ``` Prints each shard's record count/file size and whether its process is still alive, then re-derives the old table's discrete-cell coverage view from the raw rows on the fly (via `old_table_buckets.py`'s exact port of `searchsup.c`'s bucket functions) -- populated-cell count, count below the same 500-sample floor the original table's own bake used, and the sparsest populated cells (the "dusty corners" view). Unlike the old `dashboard.py`, there's no accumulated histogram to read -- this one loads the shard files directly (with `snapshot=True`: a trailing partial record from a write caught mid- flight, or the writer's own `FH_COLLECT_BUFFER_RECORDS`-sized buffered- but-not-yet-flushed tail, is expected and silently dropped, not an error -- see `load_records.py`'s own docstring) each time it's run, so it's always reading real on-disk state, just never perfectly up to the instant. **Taking a mid-run snapshot to bake a model without stopping the shards** (the other thing the old workflow supported): works the same way -- pass `--snapshot` to either `explore_and_fit.py` or `spotcheck_vs_table.py` (e.g. `python3 explore_and_fit.py runs/my_run_name --snapshot` -- note these two scripts take the actual directory, not the bare run name `dashboard.py` takes) to read `/shard_*/fhcollect_thread00.bin` with `snapshot=True`, exactly like `dashboard.py` already does. The shards keep running throughout, same as the old workflow's "last checkpoint is authoritative" model -- nothing here needs them paused. **Stopping**: `./stop_shards.sh my_run_name` -- safe anytime, kills each shard by its recorded pid; the only loss on a hard kill (vs. a clean exit) is each shard's own still-buffered (not yet flushed) tail, same `FH_COLLECT_BUFFER_RECORDS - 1`-record caveat as any other abrupt stop. **Validated mechanically** (2026-10-24): a 2-shard, 10-position, `sd 8` pipeline smoke test exercised build → launch → dashboard (mid-run-shaped coverage output, including the sparsest-cells view) → stop end to end with no issues. ### 3. Load and sanity-check ```sh cd regression_run python3 load_records.py /tmp/typhoon/ ``` Prints row counts by table/tier and a label rate, and fails loudly (nonzero exit, listed problems) on any structural issue -- out-of-range table/tier/label/moves_known values, negative gap, implausible depth. Run this on every new collection directory before trusting it further. ### 4. Explore and fit ```sh python3 explore_and_fit.py /tmp/typhoon/ ``` Engineers features (log transforms, one-hot tier/node-type, normalized ordinal, the running-counter fraction, the node-type x running-counter cross term -- see `engineer_features()`), prints a per-feature discrimination report (correlation with label, bottom/top-quartile rate), fits a standardized logistic regression per table, and prints log-loss/AUC plus a calibration table (predicted-probability bucket vs. actual rate). This is the fast, routine command to re-run after any new collection pass. ### 5. Spot-check against the existing historical table ```sh python3 spotcheck_vs_table.py /tmp/typhoon/ \ ../dual_table_run/pickles/baked_now.pkl ``` Trains both models (same as step 4), then -- using Python re-implementations of `searchsup.c`'s exact bucket functions (`old_table_buckets.py`, constants pulled directly from `chess.h`, not guessed) -- finds real rows in the fresh collection that land in the *same discrete cell* as the old table's own highest-tries cells (`tier == LEFTOVER_QUIET` for the alpha side, matching the old table's real per-move population; `tier == NO_MOVE` for the beta side, matching its whole-node-only population), and prints old-table-rate vs. new-model-mean-prediction side by side. This is the "sanity version" of the two-tier validation plan described earlier in this doc -- cheap, fast, and meant to catch a sign/scaling bug, not to be the final calibration gate (that's step 5 of the numbered Plan above, against the real multi-day run, with the **rigorous** version of this same comparison). **What a good result looks like**, from the actual 11-position `ecm_ringers.ep_` smoke run: the large majority of matched cells (30/30 found on both sides against the old table's top-30-by-tries cells) track within 1-3 percentage points of the historical rate, including several beta-side cells with billions of real historical tries. Isolated larger deviations are expected and not automatically alarming -- e.g. this run's one real outlier landed on the `late` axis's known-unusual saturated tail bucket (see "Why past LMR experiments failed"-adjacent discussion of bucket 32 earlier in this doc); a *systematic* drift across many cells in the same direction, rather than one isolated bucket, is the thing that would actually warrant stopping and investigating before trusting a model trained this way. ### 6. Quantize and bake (not yet automated -- next piece of work) Not built yet. When this is wired up, the shape is: take the fitted `sklearn` `LogisticRegression` + `StandardScaler` (as returned by `explore_and_fit.fit_and_report`), fold the scaler's mean/scale affinely into the raw-scale weights and intercept (`coef / scaler.scale_`, with the intercept adjusted by `-sum(coef * mean / scale)` -- exact algebra, no approximation, since standardization is just an affine transform), quantize to fixed-point integers at a scale chosen *after* seeing the real weight magnitudes (don't pre-commit to e.g. int16 the way the old table's axes had to -- there's no storage pressure forcing a tight scale here), and emit a small C array (weights + bias + a sigmoid lookup table) analogous to how `bake_dna.py` splices tuned constants into `eval.c` today. ### 7. Wire into the engine (not yet done -- after step 6 and the full validation ladder) Replace the `_JointTablePredictedFHThousandths`/ `_BetaTablePredictedFHThousandths` call sites in `searchsup.c` with the quantized dot-product + sigmoid-lookup inference. Concretely: - The feature vector at each call site needs to be assembled from values already computed there today (`iEval`/`iAlpha`/`iBeta` for gap, `uDepth` for depth, the existing `uRawImproveB`-style locals, the move's own tier/ordinal/`EstimateHangingMaterialCp`/SEE value, `GetMoveFailHighWilsonUpperBoundThousandths`/`GetMoveFailHighAttempts` for the evidence features) -- nothing new to compute, just reshaped into the model's input order. - `_RefineNodeTypeGuess` and the null-move-skip site both currently reuse one table lookup (`uNodeLocalFH`) across multiple call sites per node specifically to avoid a duplicate table consult -- preserve that same reuse discipline with the new inference call, which is not free (a ~20-feature dot product), unlike a bare array index. - Per the "One model, not two" decision above, there is no `g_fUseNnueEval` branch needed at the call site -- always use the one NNUE-trained model, under both HCE and NNUE builds. - Full validation ladder before this ever ships as the default: `gmake DEBUG=1 TEST=1` + `debug_smoke_test.sh`, per-move FH-rate instrumentation, `sd 10`/`sn 5000000` suite diffs on the three curated suites, `match_play.py` gated self-play (`LOWER95 >= 0.5`). See `docs/DEEPER_PRUNING.md`'s existing validation-ladder discipline -- this is the same ladder, not a new one. ## Session note: depth-dependent sampling bias (2026-10-24 follow-up) After hours of `nnue_run1` collection with tier-dependent sampling live, Scott noticed the beta-side rigorous spot-check's depth-blowup (`spotcheck_vs_table.py`, depth-bucketed error climbing from ~2pp at shallow remaining depth to 20-37pp at the deepest buckets) wasn't shrinking despite more hours of collection -- surprising, since more data usually does narrow this kind of gap. Root cause: `fhcollect.c`'s tier-dependent subsampling classified beta-side `NO_MOVE` (the whole-node population, exactly what the spot-check compares against the old table) as "common," so it got hit by the aggressive `FH_COLLECT_SAMPLE_DENOM` (e.g. 1-in-250) uniformly across ALL remaining depths. But a large REMAINING depth means the node sits close to the root, where the branching factor makes nodes structurally scarce compared to the shallow-remaining-depth population near the leaves -- the deep tail's raw volume was always small relative to the shallow glut, for tree-shape reasons having nothing to do with how long collection runs. Subsampling both populations at the same flat rate preserves that same scarcity ratio forever: running longer scales the deep tail and the shallow glut up together, so the deep tail's SHARE of the data (what the spot-check's matched-cell `n` counts depend on) never improves. This is a sampling-design bug, not evidence the model itself can't fit deep cells -- confirmed by the matched-cell `n_new` counts themselves staying in the single/low double digits at depth 5+ even after a full extra collection pass. Fix: `FhCollectEmit`'s common/rare classification is now ALSO depth-aware, not just tier-aware -- once `uRemainingDepthTicks / ONE_PLY >= 3`, a record falls back to the gentle/rare denominator regardless of tier, on both tables. Cheap to do (the deep population is naturally small on its own, so relaxing its sampling doesn't meaningfully grow disk usage) and directly targets exactly the gap the spot-check kept surfacing. **Required a full shard restart** (the denominators, and now this depth check, are read/applied once per process -- a running shard keeps using the old binary's logic until restarted), same as the original tier-dependent-sampling fix. Restarting costs nothing but some redundant re-processing of each shard's already-seen opening positions (search is deterministic, so no bias is introduced -- see that section's discussion); existing collected data is kept (append- mode writer, never truncated). Re-run `spotcheck_vs_table.py` again after this fix has had a few more hours to accumulate deep-bucket coverage before concluding whether it actually closed the gap. ## Beta-side spot-check disagreement, investigation, and resolution (2026-10-24) The rigorous spot-check (`spotcheck_vs_table.py`) consistently showed a real, unexplained disagreement between the fresh beta model and the old table, concentrated in two distinct ways at different points in the investigation -- both eventually resolved (or, for the second, found NOT to be a model problem), via direct measurement rather than guessing: **1. Mate-score window contamination (resolved).** The old table's `gapbeta_b=20` overflow bucket (gap >= 2000cp) showed the fresh model badly OVER-predicting (97-98%) against the old table's 25-58%. Checked directly: that bucket is 99.4% concentrated at `gap` in [31700, 32767] -- essentially `INFINITY_SCORE` -- with a complete gap in the data between 3,000 and 20,000, meaning it's not a continuum of "big material edge getting bigger" but two distinct populations: an ordinary cp-scale edge, and a detected-mate static eval compared against an ordinary beta window. The fresh collection's own recorded label rate for the mate-score population was high (90-99% at shallow/medium depth) -- the "disagreement" was the FRESH data being right and the OLD TABLE being stale for this one narrow, structurally different region (very likely a holdover from before the NODE_TYPE-refinement rewrite, commit `69525eb`, landed literally hours before the old table's own bake). Fixed by excluding `gap > MATE_GAP_THRESHOLD` (5000, chosen in the empirical no-data gap between the two populations) from BOTH training (`load_records.exclude_mate_window`) and the old-table comparison population (the old table's own overflow-bucket cells, since its ground truth there is the stale/mismatched side) -- see `old_table_buckets.py`'s `exclude_overflow` parameter to `spotcheck_vs_table.run_comparison`. This also directly validated Scott's independent hypothesis about `iBeta > +NMATE` windows being a structurally different regime not worth training on. **2. A second, smaller depth/improve-combo disagreement (ruled out as a model defect -- NOT the same issue).** After the mate-window fix, 3-4pp average / up to 34pp disagreement remained at moderate depth, ordinary gap, specific own/enemy-improve combos. Two hypotheses were tested and ruled out: (a) more collection volume -- tested directly by resampling 5x bigger (15M vs. 3M rows, same already-collected data, no more time running shards) and the error got slightly WORSE, not better, which is the signature of a real bias, not sampling noise; (b) a missing `improve x enemy_improve` interaction term -- added and refit, no measurable effect. The test that actually settled it (Scott's own proposed validation criterion -- "run a zillion positions and see if it approximates the predicted rate"): a targeted calibration check on EXACTLY the disputed region, held out from training, with real observed outcomes. Over 258,349 held-out rows in that region, predicted probability matched realized frequency within 0.04-3.5pp at every decile, aggregate 46.10% predicted vs. 46.85% actual. The model's predictions ARE calibrated against real outcomes in the disputed region -- the old-table disagreement there is most likely explained by the same staleness mechanism as (1) (old table built before the NODE_TYPE-refinement rewrite), not a defect in the fresh model. **Validation methodology this settled on** (the actual answer to "how do we know this model is trustworthy," distinct from "does it agree with the old table," which can be unfair when the old table itself is stale): bucket held-out rows (never used in training) by their own predicted-probability decile, and check whether the REALIZED label rate in each decile matches the mean predicted probability for that decile. This is a direct test against ground truth (what actually happened), not against a second model (the old table) that may itself be wrong. Agreement with the old table is useful corroboration when it holds, but calibration against real outcomes is the primary trust criterion, and the one that should be re-run after any future search/eval change (the same risk that made the old table stale applies to this model too, just on a longer horizon). ## Production implementation (`fhmodel.c`) and a serious open problem found by actually benchmarking it (2026-10-24) Implemented as fully quantized fixed-point integer arithmetic (Q16 for table/final-sum arithmetic, Q30 for the handful of raw-feature weights) -- explicitly NOT floats, per the original design premise ("the whole premise of this regression model is that it would be fast, have few multiplies and adds, and could be quantized" -- not optional, corrected after an initial draft wrongly used floats by analogy to NNUE's L2/L3 tail, which Scott correctly pushed back on). See `fhmodel.c`/`chess.h` (declarations folded into chess.h, matching fhcollect.c's own no-separate-header convention) and `regression_run/export_model.py` (generates `fhmodel_weights.h`). **Hoisted node-level/move-level split** (Scott: "make sure to hoist as much of the inference out of the move loop as you can... gap to alpha is one of the terms that can be hoisted"): `gap`, `depth`, `improve`, `enemy_improve`, `material`, and `node_type` are all fixed for a node's entire move loop (none can change based on which move is being tried), so `FhModelAlphaNodeBaseQ16` computes them exactly ONCE per node (search.c, same point `FhCollectNodeEntry` already is), and `SearchOneMove` only adds the genuinely move-varying terms (move value, ordinal, tier, this move's own historical FH% evidence) per move. **Production feature-set scoping**: `good_captures_tried_no_cutoff` and its cross term were dropped from the production model (though kept for data collection/analysis) -- using them live would require adding new always-on state to `search.c`'s move loop (currently collection-only), and measured cost of dropping them was negligible (AUC -0.0004/-0.0002 on alpha/beta). `FhClassifyMoveTier` was moved from `fhcollect.c` (collection-only) to `searchsup.c` (always compiled) since production inference needs real tier classification unconditionally, not the collection-only build's no-op stub. **Real bugs found and fixed via actually building and running this**, not by inspection: - The sigmoid-output lookup table overflowed `INT` (32-bit): an earlier version Q16-scaled `probability*100000` (max ~100000) on top of an already-complete integer unit, producing values up to ~6.55 billion, ~3x `INT32_MAX` -- caught via a real `-Wconstant-conversion` warning on the actual generated header, not by inspection. Fixed by not double-scaling: the table already outputs the final integer unit directly. - `_RefineNodeTypeGuess`'s alpha whole-node branch initially called a function name left over from an earlier (pre-quantization) draft that no longer existed after the rewrite -- caught by the compiler (`undeclared function`), not by review. **A serious, SERIOUS, still-UNRESOLVED problem found by running `bench`** (Scott's explicit ask: compare `bench` speed against HEAD with `git stash`): total nodes searched at the SAME fixed depths (`bench.c`'s own per-position depth, unaffected by this change) dropped from 1,678,274 (HEAD, table-based) to 302,251 (regression model) -- a 5.5x reduction, not a modest speed difference. Traced via temporary diagnostic instrumentation (added, used, and removed in this same session -- not left in the tree) directly logging `FhModelBetaPredictedFHThousandths`'s output: it returns 94.5-99.97% for EVERY sampled call, regardless of gap (9-1299), depth (1-7 plies), or improve/enemy values varying widely. This saturates `_RefineNodeTypeGuess`'s band classification into `NODE_CUT` almost universally, which cascades into far more aggressive pruning everywhere else in the engine that reads `pi->eNodeType` -- almost certainly the real cause of the 5.5x node-count collapse, not a reduced-dynamic-range-but-still-reasonable-magnitude issue. **Root cause**: `FHM_BETA_INTERCEPT` is `+5.55` in real log-odds units (`sigmoid(5.55)` ~= 99.6%) -- confirmed present even in the PRE-quantization float fit, not introduced by the integer rewrite. `export_model.py`'s `fit_final()` trains on the FULL mixed-tier beta population (every per-move tier plus `NO_MOVE`) with just an additive per-tier offset, but the only thing production ever calls (`_RefineNodeTypeGuess`) is the `NO_MOVE`-specific whole-node case. `NO_MOVE` rows are inherently high-base-rate (the node's static eval already exceeds beta by construction of eligibility), and the shared node-level slopes -- fit across ALL tiers combined, dominated by the much larger per-move-tier population's own data -- are too flat relative to that large additive offset to meaningfully move `NO_MOVE` predictions away from it. The AUC=0.9234 reported during fitting is real but measures discrimination ACROSS tiers (tier identity alone separates classes well), not WITHIN the one tier (`NO_MOVE`) production actually consumes -- exactly the kind of gap the holdout-calibration methodology above would have caught if it had been re-run against the FINAL production (trimmed-feature, mixed-population-fit) model specifically restricted to `NO_MOVE` rows, rather than the earlier, differently-fit model that calibration check used. **Not yet fixed. Recommended next step**: refit the beta production model on the `NO_MOVE`-only population specifically (not the mixed population used for data collection/the general calibration work), so the intercept and node-level slopes reflect `NO_MOVE`'s own actual relationship to gap/depth/improve/material, not an average diluted by tiers production never queries this function for. Re-run the holdout calibration check (restricted to `NO_MOVE` rows specifically) and the `bench` node-count comparison against HEAD again before trusting this number -- do NOT re-attempt a speed comparison until this is fixed, since the current 5.5x node-count gap makes any NPS number measured against it meaningless (comparing search quality/aggressiveness, not per-node computational cost). **Build profiles verified clean** (all three): default (table-based, unchanged), `FH_REGRESSION_COLLECTION=1` (collection), and the new `FH_USE_REGRESSION_MODEL=1` (this implementation) all compile with `gmake clean && gmake release PERF_COUNTERS=1 MP=1 [flag]=1` with zero errors. `gmake DEBUG=1 TEST=1 FH_USE_REGRESSION_MODEL=1`'s self-test suite ran to completion with no assertion failures or crashes (`REAL_EXIT=0`) -- the new code path is memory/assert-safe even though its PRUNING BEHAVIOR is currently wrong per the above. Do not treat "doesn't crash" as "is correct" -- they were checked separately here on purpose, and only the first currently passes. ## Trust audit (2026-10-24, same day): a close read before committing anything Scott's framing going in: the candidate won a 400-game match (54.4% was the actual score, see below -- "65%" was a misremembering, not what the log shows) and bench looked ~fixed, but the whole process "felt haphazard, seat-of-the-pants, rushed, and random," and he specifically distrusted (a) the beta-side saturation fix, (b) a verbal "the table was too conservative because of smoothing" explanation for a threshold change, and (c) a "this feature isn't a strong signal" dismissal of the good-captures-counter generalization. All three turned out to be worth distrusting, to different degrees -- see below. This section is the full audit trail; treat it as the actual state of the beta-side work, superseding the "Not yet fixed" framing a few sections up wherever they conflict. ### A real, unrelated bug found while trying to build anything The working tree's `search.c` didn't compile at all, with no FH flags present -- a prior edit had collapsed `SCORE iBestScore;` (declared near the top of `Search()`) into `SCORE iBestScore = -INFINITY_SCORE;` and left the combined declaration *below* the early-QSearch-jump that already uses `iBestScore` at line ~402. Fixed by restoring the plain early declaration and leaving the assignment where it was. Unrelated to any FH work; just a reminder that "does it build" is step zero, not an assumption. ### Match-play log: 54.4%, not 65%, and it fails the project's own gate `eval_tune/regmodel_vs_head_nnue.log` (400 games, built from binaries postdating the beta NO_MOVE-only refit) is 142W/151D/107L = **score 0.5438**. Recomputing with `match_play.py`'s own formula (plain normal-approximation CI, not Wilson): **LOWER95 ≈ 0.495** -- below the documented `>= 0.5` gate. So, as of that run, this candidate had not actually cleared the bar it needs to clear; "it won" was not an accurate read of the saved log. ### Bench: NPS is the wrong metric; total nodes told a different story `bench.c` prints `"Searched N nodes in T sec"` *before* the NPS line. Comparing NPS across builds (as a first pass did) can look flat even when the tree collapsed, because fewer nodes in proportionally less time preserves throughput. Using the actual node-count line, same fixed depths both builds: | build | nodes | ratio vs HEAD | |---|---|---| | HEAD | 1,678,274 | 1.00x | | regression (threshold=96000, the shipped default at the time) | 450,852 | **3.72x collapse** | | regression (threshold=99000) | 1,489,894 | 1.13x (close) | The original bug report's 5.5x collapse (pre NO_MOVE-only refit) had shrunk to 3.72x post-refit -- a real improvement, but "the bench issue is ~fixed" (said based on NPS alone) was not correct; the collapse was still present, just smaller, until the threshold was also addressed (see below). ### ECM suites confirm it, at real cost to tactical strength `sd 10`, three curated suites, HEAD vs regression at the then-default threshold (96000): | suite | HEAD solved | regression solved (96000) | regression solved (99000) | |---|---|---|---| | ecm_ringers (11) | 2 | 3 (noise-level, n too small) | -- | | ecm_confident_quick (90) | 56 | **41** (-15) | **56** (exact match) | | ecm_hard_quick (90) | 13 | **7** (-6) | **12** (within noise) | ### `good_captures_tried_no_cutoff`: the signal is real, not broken by the generalization Checked both the doc's claimed ablation (-0.0004/-0.0002 AUC on alpha/beta for dropping it from the production feature set -- no saved artifact existed anywhere for this number, but re-measuring it independently, same methodology, confirms it: +0.0007/+0.0001 on 4M rows, negligible either way) and the raw univariate signal: ``` good_captures_tried_no_cutoff=0 FH_rate=20.42% =1 FH_rate= 3.10% =2 FH_rate= 1.69% =3+ FH_rate~ 1.0-1.8% ``` ~15-20x effect on its own, univariate AUC=0.68. It wasn't broken by Scott's generalization from "hash move is a capture" -- it correlates +0.41 with `log_ordinal`, a feature that didn't exist in the old table's axis set and does exist in the new model, so the generalized counter's real content is now redundant with (absorbed by) `log_ordinal` in the full model. The marginal-AUC number is a true statement about *redundancy given the richer feature set*, not a true statement about *whether the phenomenon is real* -- conflating the two, especially right next to a mention of not wanting new move-loop state, is exactly what produced the dismissive-sounding "meh, not a strong signal anyway" framing. The underlying finding stands; it just doesn't need its own dedicated feature once `log_ordinal` is present. ### Calibration: the beta model is honest, both in aggregate and sliced Held-out decile calibration, NO_MOVE population, post-refit model: ``` pred [0.950,0.960) n=4,050 mean_pred=0.9549 realized=0.9533 pred [0.960,0.970) n=3,966 mean_pred=0.9649 realized=0.9652 pred [0.970,0.980) n=3,994 mean_pred=0.9751 realized=0.9735 pred [0.980,0.985) n=1,979 mean_pred=0.9825 realized=0.9843 ``` Sliced further by depth and material within the "would-skip" (pred>=0.96) population specifically (to hunt for a bad subpopulation before touching the global threshold): wrong-rate stays flat and low (0.6%-3.6%) everywhere checked, depth 1-9 plies and material 0-7000cp alike. No identifiable bad corner. **The "table was too conservative because of smoothing, the regression model is more accurate" explanation for needing `FH_BETA_SKIP_THRESHOLD=99000` is not supported by this data** -- there's no smoothing gap to correct; the model's 96% output is already honest, on average and in every slice checked. ### The real explanation: firing FREQUENCY, not per-decision accuracy Comparing the table's own real historical wrong-rate at its 96000 threshold (`dual_table_run_nnue/pickles/baked_now.pkl`'s raw NO_MOVE cells, >=50,000 tries each, mate-window excluded) against the regression model's held-out wrong-rate at the same nominal threshold: | | wrong rate @ 96% | |---|---| | old table (historical, 66B real tries) | 1.77% | | regression model (held-out) | 2.02% | Close -- the two models roughly agree on what "96%" means in real risk, aggregate. So the ECM/bench collapse is not explained by the regression model being generically overconfident at 96% relative to the table. Added a small diagnostic (`u64QuickNullMoves`/`u64QuickNullMoveSuccess` tree counters, already existed; wired through to `g_Options` in `root.c`, accumulated across a suite in `script.c`'s `g_SuiteCounters.u64TotalQuickNullMove{s,Success}`, printed as a new "quick nullmove fire" line in `_PrintSuiteReport` -- kept, not a remove-after-session diagnostic, since it's cheap and generally useful) and re-ran the suites: | suite | HEAD fire rate | regression fire rate (96000) | regression fire rate (99000) | |---|---|---|---| | ecm_confident_quick | 47.09% (714K/1.52M) | 91.61% (745K/813K) | 71.05% (1.31M/1.84M) | | ecm_hard_quick | 40.83% (833K/2.04M) | 87.49% (800K/914K) | 64.74% (1.45M/2.23M) | The regression model fires the shortcut on roughly **twice** as many eligible nodes as the table did, at a similar per-decision risk level. Since each firing truncates a subtree, a ~2x per-node firing-rate increase compounding across depth plausibly explains the 3-4x total node collapse and the resulting solve-rate cost, without requiring either model to be locally miscalibrated anywhere checked. **Working hypothesis, not yet verified**: real search trees have correlated neighboring nodes (siblings/descendants down one line often share whatever feature combination drove a high prediction -- a recent capture's eval margin, say -- for several consecutive plies). The regression model's richer feature set can track that persistent condition precisely and fire at every one of those correlated nodes; the table's coarser, smoothed buckets more often scatter similar-but- not-identical real positions across different cells, diluting the same real condition below its own 96% cutoff more often. Same honesty per isolated decision, different firing frequency once correlated decisions down a real line are strung together. Worth checking directly (e.g. autocorrelation of the shortcut firing along a single PV) before leaning on this further. ### Category A vs B: ECM was the wrong instrument to pick the threshold with `docs/DEEPER_PRUNING.md`'s framework: Category A (tighten safe-by- construction machinery, validate with FH-rate/suite-diff) vs Category B (deliberately accept a nonzero error rate for aggregate Elo, validate *only* with self-play -- "the FH-rate harness tells you where the errors are, but only self-play can tell you whether shipping them is a net win"). Tuning `FH_BETA_SKIP_THRESHOLD` to recover ECM solve parity (as this session initially did) applies the Category A instrument to what is actually a Category B decision -- ECM puzzles are curated worst-case tactical positions, exactly the population a deliberately more aggressive pruner is expected to cost solves on even when it's a real net win in ordinary games (cf. Crafty's own `search.c` comment on its forward pruning: "this is classic forward-pruning and can certainly introduce errors into the search. However, cluster testing has shown that this improves play in real games"). Given typhoon searches ~800x more nodes than Stockfish at comparable depth (`DEEPER_PRUNING.md`), the regression model's higher firing rate is not obviously a regression to correct back toward the table's rate -- it might be exactly the direction needed, and ECM solve-count alone cannot distinguish those two possibilities. The one artifact that bore on this before the sweep below was the match-play log quoted earlier, at the *aggressive* default (96000, 2x table firing rate) -- inconclusive on its own (LOWER95≈0.495). **Sweep run, 2026-10-24 (same day): 96000 vs 99000, head-to-head.** 200 games, `sd 8`, `twic_filtered.pgn` openings, both sides the regression build differing only in `FH_BETA_SKIP_THRESHOLD` (two tiny shell wrappers setting the env var, passed to `match_play.py` as the two engine paths -- no code changes needed to run this). Result, 99000 as candidate against 96000 as head: ``` CANDIDATE_SCORE=0.8425 GAMES=200 WINS=141 DRAWS=55 LOSSES=4 ELO=+291.3+/-66.1 LOWER95=0.7920 TC=sd8 nodes: candidate(99000)=1,239,521,567 head(96000)=468,297,174 ``` Not a marginal result -- `LOWER95=0.79` clears the `>=0.5` gate by a wide margin, and 96000 lost decisively despite searching *fewer* total nodes and running at higher aggregate NPS (i.e. it wasn't losing because it did less work overall in wall-clock terms; it was losing because the extra static-shortcut firings were cutting real signal, not dead weight). This settles the Category A/B ambiguity raised above: for this particular knob, the self-play result and the ECM-based intuition agree once the right instrument is actually used -- 96000's ~2x-table firing rate is a real strength cost, not a Crafty-style acceptable trade-off. **`FH_BETA_SKIP_THRESHOLD`'s default (99000, `search.c`) is now confirmed by self-play, not just a conservative stand-in.** **Follow-up sweep, same day: 99500 vs 99000.** Same setup (200 games, `sd 8`, `twic_filtered.pgn`), 99500 as candidate against 99000 as head: ``` CANDIDATE_SCORE=0.5650 GAMES=200 WINS=64 DRAWS=98 LOSSES=38 ELO=+45.4+/-48.6 LOWER95=0.4963 TC=sd8 nodes: candidate(99500)=976,833,926 head(99000)=683,750,951 ``` Inconclusive -- `LOWER95` sits just below the `>=0.5` gate, and the error bar (+/-48.6 Elo) is bigger than the point estimate. Slight nominal edge toward 99500, not a confirmed win. Combined picture across all three thresholds tested: a real cliff between 96000 and 99000 (+291 Elo, decisive), then a flattening -- consistent with a model that's genuinely miscalibrated-in-firing-frequency somewhere below ~98-99% and roughly fine above that. **No case for moving off 99000 based on this data**; pushing further (99500+) is unproven, not disproven -- would need a much bigger or SPRT-gated sample to resolve a ~45-Elo effect with this much game-to-game variance, and isn't worth prioritizing over finishing the validation ladder. ### Queued: regression-model vs HEAD, both under NNUE eval (overnight, external) Every match-play comparison so far was threshold-vs-threshold *within* the regression build -- useful for tuning the knob, but it never actually compared the regression model against the thing it's meant to replace (the `data.c` tables) under the real eval the collection was done against. Queued to run overnight, outside this session: ```sh cd /usr/home/scott/typhoon/src/eval_tune python3 match_play.py \ /tmp/typhoon/threshold_sweep/typhoon_head_nnue.sh \ /tmp/typhoon/threshold_sweep/typhoon_regmodel_nnue.sh \ --pgn ../../pgn/twic_filtered.pgn \ --games 20000 \ --workers 18 \ --sd 8 \ --sprt --elo0 0 --elo1 5 \ --scratch /usr/local/tmp/typhoon_match_regmodel_vs_head_nnue_overnight \ --log /tmp/typhoon/threshold_sweep/regmodel_vs_head_nnue_overnight.log ``` Both sides run under NNUE eval (thin wrappers at `/tmp/typhoon/threshold_sweep/typhoon_{head,regmodel}_nnue.sh`, each setting `TYPHOON_BIN` and exec'ing `src/typhoon_nnue`'s existing `--nnueweights`-injecting shim against `src/eval.nnue`, the same promoted-current net both ways) -- isolates the FH-model question (table vs. regression) from the eval question, which is the actual thing this match needs to answer. `typhoon_regmodel` carries the confirmed 99000 default. `--sprt` so an unattended overnight run stops on its own rather than always paying for a fixed game count regardless of effect size. Result pending; fold into this doc once it lands. ### Scope note: the per-move beta prediction was lost, not just weakened Checked directly: `FhModelBetaPredictedFHThousandths` has exactly one call site in production (`searchsup.c`'s `_RefineNodeTypeGuess`, whole-node branch). There is no per-move beta call anywhere in `search.c`'s move loop. The original motivation for this whole model -- getting an FH% estimate on individual candidate moves when `eval >= beta` and null-move didn't resolve things, to drive pruning on specific moves beyond the null-move probe -- isn't implemented at all. The NO_MOVE-only refit fixed the saturation bug by narrowing scope down to exactly the one thing still wired in, not by fixing the underlying shared-population-fit problem in a way that would let a per-move beta model come back. **Reframing, 2026-10-24 (Scott + this session)**: given the Category A/B discussion above, this may not be worth re-pursuing as originally scoped. The whole-node-only model that actually got built is already a deliberately-unsound, Crafty/Stockfish-style forward-pruning layer in spirit (predict the node's own FH% once, before the move loop, and skip it outright on a high-confidence prediction) -- rather than treating the missing per-move/post-null-move case as a gap to backfill, it may be more productive to lean into the whole-node framing that already exists, validate its firing-rate/threshold choice properly via self-play (above), and revisit the harder per-move case only if that doesn't pan out. ### Alpha-side trust, quantitatively confirmed Re-ran `spotcheck_vs_table.py` against the real NNUE table (`dual_table_run_nnue/pickles/baked_now.pkl`), full rigor (old_tries >= 50,000 per cell, not the earlier small smoke sample): - **Alpha**: weighted mean |old_rate - new_rate| = **0.119pp**, median 0.037pp, across 10,684 matched cells. Worst disagreements (up to 7.6pp) are all in single-digit-`n_new` deep-depth cells -- sampling noise, not a trend. This confirms, with real numbers, Scott's standing trust in the alpha-side model: it tracks the old table closely almost everywhere there's real data on both sides. - **Beta** (this same run, using `explore_and_fit`'s general mixed-tier fit, i.e. the *pre-refit* shape): 3.528pp weighted mean, growing to 20-30pp at depth -- reproduces the already-diagnosed saturation problem through a script path (`spotcheck_vs_table.py`) that was never updated to use the NO_MOVE-only refit. Not new information, but a loose end: that script should be updated to match `export_model.py`'s actual production fit before it's used as a beta-side trust check again. ## Explicitly out of scope for this pass - PV-node collection/modeling (real `alpha < eval < beta` states) -- future work, not attempted here. - Any model family beyond logistic regression for the production path -- trees and nets were considered and rejected on inference-speed grounds for this specific inner-loop use; revisit only if logistic regression's calibration proves insufficient after real collection.