Phase A NNUE: feature-indexing spec, annotation/training pipeline, dump nnuefeatures

Builds out the NNUE Phase A prototype per docs/NNUE.md:

- src/nnuefeatures.c: standalone HalfKAv2_hm feature-index reference
  implementation (new `dump nnuefeatures` command), sourced from
  Stockfish's half_ka_v2_hm.{h,cpp} (sf_16 tag), including the 0x88->64
  square conversion typhoon's own board representation needs.
- annotate_pgns/: sharded TWIC/self-play annotation harness
  (annotate.py), position extraction, quiet-position filtering (flat
  and game-shaped, plus an upstream --prefilter-quiet option),
  train/holdout game-level stratified split, WDL-sigmoid K fitting,
  the Python HalfKAv2_hm reference encoder (nnue_feature_spec.py) and
  its bit-for-bit cross-checker against the C side
  (cross_check_features.py), and a first PyTorch training pipeline
  (nnue_model.py, nnue_train.py, build_training_tensors.py) -- a
  256x2-32-32-1-ish net trained locally on CPU against ~1.85M quiet
  TWIC positions as a pipeline/framework validation pass.
- Finish an in-progress InitMagic/InitLMRTable/InitEval ->
  InitializeBitboardMagic/InitializeLMRTable/InitializeEval rename:
  chess.h's prototypes hadn't been updated to match the already-renamed
  definitions/call sites, leaving 4 implicit-function-declaration
  warnings.
- tests/arasan.epd: new curated EPD suite.
- docs/: add the project's design docs (previously untracked).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 234 insertions, 0 deletions

diff --git a/src/docs/EFP_LMP_MERGE.md b/src/docs/EFP_LMP_MERGE.md
new file mode 100644
index 0000000..e2bc27a
--- /dev/null
+++ b/src/docs/EFP_LMP_MERGE.md
@@ -0,0 +1,234 @@
+# Retiring EFP in favor of LMP
+
+Status: **plan only, not started.** Supersedes item 4 ("EFP-vs-LMP
+substitution experiment") in `EXTENDED_PRUNING_DEPTH.md`, which flagged this
+as the next step but didn't have the concrete numbers below. Do not start
+until the current uncommitted working-tree changes (presently being
+evaluated against HEAD via `match_play.py`, score trending positive) are
+either committed or reverted -- this work should start from a clean,
+committed baseline, not stack on top of an in-flight, unresolved candidate.
+
+## Motivation
+
+EFP (`search.c`'s `fTryEFPHere`/`fThisMoveEFPPruned`, ~line 1440-1522) and
+LMP (`ShouldLatePrune`, `searchsup.c`) are two separate forward-pruning
+mechanisms that fire on heavily overlapping populations at remaining depth
+0-2 ply, maintained independently, with no shared tuning discipline between
+them. EFP uses a hand-picked margin (Heinz's paper: `VALUE_KNIGHT`/
+`VALUE_ROOK` by 2-tier depth) plus two fail-high-history vetoes (static
+table, dynamic per-move). LMP uses the same underlying 5D joint fail-high
+table (`g_uJointFHThousandths`, `data.c`) directly, with no margin test and
+its own, differently-calibrated pair of vetoes. Goal: retire EFP entirely,
+let LMP's table-driven decision govern that depth range on its own, without
+losing EFP's pruning volume (tree size must not grow) and without adopting
+Crafty's `break`-the-whole-node semantics (every mechanism here stays
+per-move, `continue`-based -- rejected explicitly, see "What we are
+deliberately not doing" below).
+
+## What's already been verified (no code changes made yet)
+
+1. **The joint table has no sentinels.** `g_uJointFHThousandths` (`data.c`)
+   is fully backoff-smoothed (Wilson-upper-bound, shrinkage to coarser
+   marginals) as of the mid-run splice documented in
+   `../dual_table_run/README.md` -- "No 0xffff sentinels remain." Both
+   `ShouldLatePrune`'s and EFP's sentinel checks are currently dead code,
+   not a live concern.
+
+2. **The table's collection was censored by EFP/LMP themselves, and that's
+   expected, not a flaw.** `search.c`'s `FH_TABLE_COLLECTION` instrumentation
+   sits textually *after* the `if (fThisMoveEFPPruned || fThisMoveLMPPruned)
+   { ...; continue; }` block (~line 1550-1613) during the collection run
+   that built this table. A move either mechanism decided to prune never
+   reached the collection counters. This means the table is accurate for
+   the population it actually saw (eligible moves that survived both
+   mechanisms) but has **zero data** on the population EFP has been
+   discarding -- not biased data, no data. There is no way to query "what
+   was the real FH% of EFP's historically-pruned moves" from existing logs;
+   it was never measured. (Corrects an earlier, imprecise claim in this
+   investigation that bucket[0]/[1] were "thin" there -- they aren't thin,
+   they're simply about a different population.)
+
+3. **EFP's own margin, read through our own table, is already far more
+   conservative than its nominal static-veto bar.** Looking up EFP's tier-1
+   margin (`VALUE_KNIGHT`≈325cp, depth bucket 1, gap bucket 4) in the live
+   table gives a measured FH% of 0.01-0.05% across lateness buckets --
+   roughly 10x *under* EFP's own `uPredictedFH < 300` (0.3%) static veto.
+   That veto has essentially never been the operative gate; the margin test
+   alone already does the real filtering, more conservatively than its own
+   stated threshold.
+
+4. **LMP's existing static ceiling already has slack to cover EFP's old
+   population, unmodified.** Comparing EFP's old population's real
+   table-measured FH% against LMP's current `uLMPStaticMaxPctThousandths[]`
+   ceiling by depth bucket:
+
+   | depth bucket | LMP's current ceiling | EFP's-old-population measured FH% |
+   |---|---|---|
+   | 0 | 0.200% | 0.010% |
+   | 1 | 0.200% | 0.020% |
+   | 2 | 0.150% | 0.036% |
+   | 3 | 0.075% | 0.063% |
+   | 4 | 0.050% | 0.071% (now exceeds ceiling -- correct, by design) |
+
+   At depth buckets 0-2 (EFP's actual operating range), LMP's own ceiling
+   already permits 5-20x more risk than EFP's population actually carries.
+   The static side of LMP should, on its own, already prune *at least* as
+   much as EFP did there. By depth bucket 3+, LMP's own schedule correctly
+   starts refusing -- that tightening-with-depth behavior is intentional,
+   not a gap to fix.
+
+5. **The real asymmetry is the DYNAMIC veto, not the static one, and this
+   is the mechanism most likely to cause a tree-size regression on naive
+   retirement.** Both EFP and LMP gate on real, per-move-identity observed
+   fail-high history in addition to the static table, but at wildly
+   different strictness:
+
+   | | min samples | max tolerated dynamic FH% |
+   |---|---|---|
+   | EFP (`search.c`) | 5 | **10%** (`EFP_FH_PRUNE_THRESHOLD_PCT_THOUSANDTHS=10000`) |
+   | LMP (`searchsup.c`) | 1 | **0.001%** (`LMP_DYNAMIC_MAX_PCT_THOUSANDTHS=1`) |
+
+   A 10,000x gap. `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS=1` is, in practice, a
+   "this move identity has never once failed high, for any nonzero sample
+   count" gate -- not a calibrated percentage. Any move identity with real,
+   nonzero fail-high history anywhere in the tree (common for moves tried
+   more than a handful of times) gets exempted from LMP pruning even when
+   the static table is fully confident. EFP rarely hit this problem because
+   its own dynamic bar (10%) almost never tripped. **Predicted outcome if
+   EFP is retired without also touching this constant: LMP's static side
+   alone is not sufficient to match EFP's old pruning volume, because this
+   gate silently vetoes a meaningful fraction of what the static table
+   would otherwise allow.** This is the paper's central, falsifiable
+   prediction -- see Plan step 2.
+
+6. **The apparent "Crafty's ply-2 margin predicts a LOWER fail-high rate
+   than ply-3, despite a smaller margin" oddity is a bucket-resolution
+   artifact, not a real inversion.** Crafty's 150cp (remaining ply 2) and
+   200cp (remaining ply 3) both land in our 100cp-wide gap bucket 2.
+   Holding gap bucket fixed and varying depth shows the expected
+   monotonic-with-depth increase (0.039% -> 0.053% -> 0.089% -> 0.176% ...
+   at gap bucket 2, late bucket 4, dumb=0, improve=neutral) -- "more depth
+   remaining = more chance to recover" holds throughout; our 100cp gap
+   bucket is just too coarse to resolve Crafty's 50cp margin steps at the
+   shallow end. Not something this plan needs to fix to proceed, but worth
+   remembering if a future finer-grained gap axis is ever considered.
+
+7. **Reference points from Crafty/Stockfish, translated into our table
+   (dumb=0, improve=neutral, several lateness buckets), for calibration
+   context only -- not adopted as targets:**
+   - Crafty's `FP_margin[]` at remaining ply 1 (100cp) implies **0.2-3.2%**
+     FH% in our table depending on lateness -- the single largest
+     "implies more room exists" data point of this whole investigation,
+     concentrated at the shallowest depth and earliest move numbers.
+     Crafty's deeper tiers (150-400cp, ply 2-6) land close to our own
+     existing 0.05-0.3% range -- roughly agreeing with us already.
+   - Stockfish's quiet-move futility margin translates to **0.01-0.13%**
+     throughout -- more conservative than Crafty in our terms, not less;
+     consistent with SF leaning on other layered mechanisms (move-count
+     quiet-move skip, continuation-history pruning, SEE-based pruning) to
+     do work Crafty's single margin does alone. SF's own margin test is
+     therefore not a fair apples-to-apples number by itself.
+   - Neither engine's number is being proposed as a direct target --
+     they're priors for where real engines have found a workable
+     risk/reward point, used to sanity-check, not dictate, what we test.
+
+## What we are deliberately not doing
+
+**No `break`.** Crafty `break`s its entire move loop on either its
+futility or LMP check -- abandons every remaining move at that node, not
+just the one under consideration. Stockfish's futility tests are `continue`
+(per-move, like ours); its one break-like mechanism (`skip_quiet_moves()`)
+is narrower still -- it only stops trying further *quiet* moves, captures
+and checks still get considered afterward. Both EFP and LMP already use
+`continue`, evaluating every move's prune/no-prune decision independently,
+and that is staying exactly as-is. Nothing in this plan should introduce a
+node-wide bailout of any kind.
+
+## Plan
+
+1. **Baseline.** Confirm `head_reference/` is current at a clean, committed
+   HEAD (rebuild if the in-flight uncommitted changes mentioned above get
+   committed first). Capture `sd 10` and `sn 5000000` logs for all three
+   curated suites (`ecm_ringers`, `ecm_confident_quick`, `ecm_hard_quick`)
+   if not already fresh.
+
+2. **Two-variant node-count experiment, before touching `match_play.py` at
+   all.** Build and compare two variants against the baseline, fixed `sd`,
+   all three curated suites, node count + first-move-cutoff-rate as the
+   signal (per CLAUDE.md's existing move-ordering-experiment methodology --
+   measure the pick, not the game):
+   - **Variant A**: delete EFP (`fTryEFPHere`, `fThisMoveEFPPruned`, and its
+     contribution to the final prune-or-continue check) entirely.
+     `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS` left untouched at `1`.
+     **Predicted result**: tree size grows relative to baseline, confirming
+     finding 5 above -- the dynamic veto asymmetry is real and costs
+     pruning volume LMP's static side alone doesn't recover.
+   - **Variant B**: same EFP deletion, plus raise
+     `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS` toward EFP's old bar (start at
+     `10000` to match EFP exactly, as the cleanest single-variable swap;
+     narrower values can be swept later if this overshoots).
+     **Predicted result**: tree size lands close to or tighter than
+     baseline, since finding 4 already showed the static side has headroom
+     once the dynamic gate stops over-vetoing.
+   - If both predictions hold, that's strong, cheap (no `match_play.py`
+     games needed yet) confirmation of the mechanism before spending any
+     self-play budget. If either prediction is wrong, stop and re-diagnose
+     before proceeding -- a wrong prediction here means the mental model of
+     which gate is load-bearing is incomplete, and tuning further on top of
+     a wrong model wastes the expensive match-play stage.
+
+3. **`match_play.py` gate on Variant B only** (not A -- A is a diagnostic,
+   not a candidate; it's not expected to be an improvement on its own).
+   No cheap/fast elimination stage exists in this project's tooling --
+   there's no shortcut cheaper than a full run, so don't plan around one.
+   Use step 2's node-count + fail-high-rate-by-bucket diagnostic as the
+   only pre-filter (lean on it harder than a minimal single-value check --
+   compare across all three suites and several buckets, not just
+   aggregate node count, since it's the last free signal before paying
+   for self-play). Then commit to **one full-length, long-run
+   `match_play.py` game count** (matching the run already used for the
+   current unrelated candidate) directly on Variant B at
+   `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS=10000` -- don't pre-plan a sweep.
+   Gate on `LOWER95 >= 0.5`, not raw score, per existing project
+   convention. Only spend a second full run sweeping a narrower
+   `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS` value if this first run comes back
+   close-but-not-passing; if it misses badly, that's a signal the
+   dynamic-veto model itself is off (go to step 5), not that a nearby
+   value would have worked.
+
+4. **If Variant B wins**: this becomes the EFP-retirement commit. Remove
+   the now-dead EFP code paths, `EFP_FH_MIN_SAMPLES`/
+   `EFP_FH_PRUNE_THRESHOLD_PCT_THOUSANDTHS` macros, and update
+   `ShouldLatePrune`'s docstring again to drop the "second gate, applied
+   after EFP" framing (it's no longer applied after anything -- it's the
+   only gate). Rebuild `head_reference/` as the new baseline.
+
+5. **If Variant B does not win** (tree size recovers per the prediction but
+   playing strength doesn't hold, or the dynamic-veto fix overshoots into
+   real tactical misses): fall back to sweeping
+   `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS` between `1` and `10000` rather than
+   either extreme, same one-variable-at-a-time discipline, before
+   concluding EFP's margin test was catching something the table's axes
+   genuinely can't see (in which case EFP's own tiers should be recalibrated
+   against measured surprise rate instead of retired, per
+   `EXTENDED_PRUNING_DEPTH.md`'s original fallback plan).
+
+## Open items, not part of this plan's critical path
+
+- **Horizon-tier loosening beyond matching EFP** (bucket 0/1's static
+  ceiling raised past its current 200/1000, toward Crafty's own
+  ~1-3%-at-early-lateness risk tolerance) is a separate, later experiment --
+  only worth running once EFP retirement itself is confirmed at parity.
+  Don't conflate "retire EFP without regressing" with "also get more
+  aggressive than EFP ever was" in the same test; they have different,
+  separable risk profiles and should be gated independently.
+- **`split.c` has neither EFP nor LMP today** (confirmed: no
+  `fTryEFPHere`/`fThisMoveEFPPruned`/`ShouldLatePrune` references in
+  `split.c`). This plan doesn't change that; `UNIFY_SEARCH.md` already
+  tracks bringing LMP (and, after this plan, LMP-without-EFP) to the split
+  path as its own separate work.
+- **Gap-bucket resolution** (currently 100cp-wide, per finding 6) is coarser
+  than Crafty's own margin schedule moves in. Not a blocker for this plan,
+  but flagged for whoever eventually tackles the horizon-tier-loosening
+  follow-on above, since a coarse bucket there would blur exactly the
+  region that follow-on cares about most.