Phase A NNUE: feature-indexing spec, annotation/training pipeline, dump nnuefeatures

Builds out the NNUE Phase A prototype per docs/NNUE.md:

- src/nnuefeatures.c: standalone HalfKAv2_hm feature-index reference
  implementation (new `dump nnuefeatures` command), sourced from
  Stockfish's half_ka_v2_hm.{h,cpp} (sf_16 tag), including the 0x88->64
  square conversion typhoon's own board representation needs.
- annotate_pgns/: sharded TWIC/self-play annotation harness
  (annotate.py), position extraction, quiet-position filtering (flat
  and game-shaped, plus an upstream --prefilter-quiet option),
  train/holdout game-level stratified split, WDL-sigmoid K fitting,
  the Python HalfKAv2_hm reference encoder (nnue_feature_spec.py) and
  its bit-for-bit cross-checker against the C side
  (cross_check_features.py), and a first PyTorch training pipeline
  (nnue_model.py, nnue_train.py, build_training_tensors.py) -- a
  256x2-32-32-1-ish net trained locally on CPU against ~1.85M quiet
  TWIC positions as a pipeline/framework validation pass.
- Finish an in-progress InitMagic/InitLMRTable/InitEval ->
  InitializeBitboardMagic/InitializeLMRTable/InitializeEval rename:
  chess.h's prototypes hadn't been updated to match the already-renamed
  definitions/call sites, leaving 4 implicit-function-declaration
  warnings.
- tests/arasan.epd: new curated EPD suite.
- docs/: add the project's design docs (previously untracked).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 226 insertions, 0 deletions

diff --git a/src/docs/MOVE_ORDERING_IDEAS.md b/src/docs/MOVE_ORDERING_IDEAS.md
new file mode 100644
index 0000000..5c65934
--- /dev/null
+++ b/src/docs/MOVE_ORDERING_IDEAS.md
@@ -0,0 +1,226 @@
+# Move ordering experiment backlog
+
+Goal: a leaner tree (fewer nodes for the same search quality) via move-ordering
+changes, not eval or pruning changes. Methodology: per CLAUDE.md's "Dynamic
+move ordering experiments" section -- measure fixed-`sd` node count /
+first-move-cutoff-rate on the three curated suites (`ecm_ringers`,
+`ecm_confident_quick`, `ecm_hard_quick`) as the primary signal, cross-checked
+against a larger `cross_sectional_200k.ep_` sample at fixed `sn` as a tertiary
+aggregate sanity check. Avoid adding throwaway per-node instrumentation code
+to the tree; prefer compile-time diagnostic swaps (like the old
+zero-selection-budget trick) that are never committed, or just the suite-level
+before/after comparison.
+
+Status key: TODO / IN PROGRESS / DONE (kept/discarded) / DROPPED (not pursuing)
+
+## Queue
+
+2. **TODO -- Qsearch check generation.** Crafty's `quiesce.c` calls
+   `GenerateChecks()` (movgen.c) to extend qsearch's first ply with
+   non-capture checking moves (direct + discovered, via attack-set
+   bitboard tricks), not just captures. Confirmed (per CLAUDE.md) our
+   QSearch is capture-only. This is a search-shape change more than a pure
+   ordering change -- re-confirm current QSearch behavior in search.c
+   before building, since CLAUDE.md notes may be stale.
+
+5. **TODO -- Countermove fallback key (coarse, empty-slot-only).** Today
+   the countermove table is keyed by the *exact* previous move; if that
+   exact key's slots are empty/unevidenced, no countermove signal fires at
+   all for that node. Add a second, coarser table keyed by
+   `[piece-type][to-square]` of the previous move, consulted *only* when
+   the exact-match slots don't fire -- a genuine fallback, not a
+   competing signal. Different from the earlier "scaled continuation-
+   history" experiment (flat across a 256x range), which tested coarse
+   keying as an always-active competing signal, conflating key-coarseness
+   with "always on." This isolates key-coarseness alone, active only in
+   the exact-match's current blind spot.
+
+## Dropped
+
+6. **DROPPED -- Flee-to-safety revival.** `FLEE_BONUS`/the commented-out
+   `cEnprise`-match code in `movesup.c`'s `SelectBestWithHistory` (and
+   the matching `RETIRED` comment in `generate.c`) was dead -- retired
+   after the unconditional GOOD_MOVE-tier version it replaced was shown
+   (evidence-calibration, pre-dates this backlog) to perform at the
+   plain-leftover baseline for the vast majority of its population.
+
+   Reran with real evidence-bucket resolution (`MOVE_ORDERING_EVIDENCE_
+   COLLECTION`, reusing `CM_EVIDENCE_BUCKETS`/`_CMEvidenceBucket` --
+   same evidence scale as countermove, no butterfly-only workaround
+   needed since this legitimately reuses the ply-1-keyed continuation
+   table unlike move-pair/recapture): `evidence >= 10000` hit 56.32%/
+   62.84%/68.23% FH (ecm_ringers/ecm_confident_quick/ecm_hard_quick)
+   vs. a ~1.35-1.54% unprivileged-leftover baseline -- comparable to
+   or exceeding countermove's own calibrated bar. A real signal the
+   prior session's cruder zero/nonzero split never isolated.
+
+   Implemented as a `GOOD_MOVE`-tier escape at that threshold, same
+   shape as `COUNTERMOVE_TIER_BONUS`. Result: **no measurable effect
+   either way** -- 0.00%/-0.05%/0.00% node delta at sd10, zero solve
+   diffs on any of the three suites, sn10M depth/cutoff flat. Not a
+   magnitude-tuning problem like move-pair/recapture had -- the
+   qualifying population (en-prise origin AND evidence >= 10000) was
+   just too rare (2,444-9,209 tries across an entire suite run, vs.
+   countermove's tens-to-hundreds-of-thousands) for any mechanism
+   acting on it to move the aggregate needle. `u64FleeTries/FH` and
+   the scoring/precompute code were fully reverted.
+
+4. **DROPPED -- Recapture bonus.** A capture landing on the same square
+   the opponent's last move landed on (`mv.cTo == mvLast.cTo`), nudged
+   within the winning/even-capture tier (`generate.c`'s `_ScoreAllMoves`,
+   `s >= 0` branch) rather than escaping it.
+
+   Built the comparison harness properly this time: `RECAPTURE_BUCKETS`
+   (2: SEE<0 losing, SEE>=0 even/winning) plus a parallel
+   `u64CaptureBaselineTries/FH` logging *every* capture (not just
+   recaptures) at the same SEE-sign buckets, so the recapture rate
+   could be read against an apples-to-apples baseline instead of in
+   isolation -- exactly the comparison move-pair's harness lacked at
+   first. Real signal: at SEE >= 0, recapture FH% ran 64-65% vs. a
+   49-52% all-captures baseline, consistent across all three curated
+   suites, on samples of 300K-4.4M tries.
+
+   Two magnitudes tried as a same-tier nudge, neither produced a
+   tree-size win:
+     - `RECAPTURE_BONUS = 50` (first guess, scaled like FLEE_BONUS's
+       old magnitude): -0.74%/+5.04%/-0.45% nodes (mixed), but a
+       **confirmed real tactical loss** -- ECM.735 (see `../tests/ECM.md`'s
+       Legit table: "HEAD's own line is clean and solid"), on
+       `ecm_hard_quick` (17->14/90 solved). Root cause, found by
+       actually computing the surrounding arithmetic rather than
+       guessing: the branch's real MVV/LVA differentiator --
+       `PIECE_VALUE_OVER_100(captured) - PIECE_VALUE_OVER_100(mover)`,
+       pawn=1/knight,bishop=3/rook=5/queen=9 -- only spans ~112-128
+       (a ~16-unit band) across *every* capture combination. 50 is
+       over 3x that whole band -- not a tiebreak, a near-dominant
+       override capable of vaulting a recapture ahead of an
+       objectively higher-MVV/LVA non-recapture capture at the same
+       node, which is exactly what cost ECM.735.
+     - `RECAPTURE_BONUS = 4` (rescaled inside the ~16-unit band):
+       fixed the regression (ECM.735 recovered, 17/90 again; the
+       hard_quick loss/gain pair that remained, ECM.491/ECM.443, are
+       both independently classified Flaky in `ECM.md` -- net real
+       change zero) but still no node-count win: +2.23%/+7.95%/-0.81%,
+       worse on `ecm_confident_quick` than the broken 50 value was.
+   sn10M depth-reached flat to within noise at both magnitudes
+   (14.76 vs. flat-history-only's 14.83), same as every move-pair
+   variant.
+
+   Conclusion: same shape as move-pair -- a real, well-calibrated FH%
+   edge that MVV/LVA's existing ordering apparently already captures
+   most of, with no tried mechanism converting the residual signal
+   into a leaner tree. `RECAPTURE_BUCKETS`/`u64RecaptureTries/FH`/
+   `u64CaptureBaselineTries/FH` and the scoring-branch code were fully
+   reverted; `MOVE_ORDERING_EVIDENCE_COLLECTION` infrastructure kept.
+
+   **Methodology note for next time**: cross-referencing actual solve
+   losses/gains against `../tests/ECM.md`'s per-position Legit/Flaky
+   classifications (rather than reading raw suite-level solve-count
+   deltas) is what separated "real tactical cost" (ECM.735) from "suite
+   noise" (ECM.491/443/496) here -- worth doing by default for any
+   future candidate's solve-count changes, not just when something
+   looks surprising.
+
+3. **DROPPED -- Ply-2 move-pair table.** Crafty maintains a discrete
+   2-slot table (`move_pair`, history.c) keyed on the *side's own* move
+   two plies back (`curmv[ply-2]`), tried in `next.c` right after the
+   countermove phase. Implemented as `mvMovePair` (same shape/indexing
+   as `mvCounter`), evidence-gated same as countermove.
+
+   First attempt (borrowed countermove's `g_HistoryCounters +
+   g_ContinuationHistory` evidence sum verbatim) measured a complete
+   no-op: identical node counts to flat-history-only across all three
+   curated suites at sd10, despite the table itself populating and
+   predicting fine (55-70% raw hit rate). Root cause: `g_ContinuationHistory`
+   is only ever written keyed by the move *one* ply back -- there's no
+   table anywhere keyed by two-plies-back, so the borrowed slot read
+   unrelated noise. Fixed to use butterfly history (`g_HistoryCounters`)
+   alone.
+
+   Built a dedicated evidence-bucket harness (`MOVE_ORDERING_EVIDENCE_
+   COLLECTION` build flag, independent of `PERF_COUNTERS` per later
+   request; `MP_EVIDENCE_BUCKETS`/`_MPEvidenceBucket`, denser floors
+   than countermove's own since butterfly-only evidence is capped at
+   `HISTORY_MAX`) and calibrated a real threshold from it: top band
+   (evidence >= 6000) hit 53.41%/64.43%/66.23% FH (ecm_ringers/
+   ecm_confident_quick/ecm_hard_quick) on large samples (26K/177K/271K
+   tries) -- comparable to or exceeding countermove's own calibrated
+   bar. A real, strong, well-sampled signal.
+
+   Three mechanisms tried to act on that calibrated signal, none
+   produced a tree-size win at sd10 across the three curated suites:
+     - GOOD_MOVE-tier escape (same pattern as countermove's own
+       promotion): +1.6%/+1.4%/+1.5% nodes -- small but consistent
+       regression across all three suites.
+     - Same-tier nudge at FLEE_BONUS's old magnitude (100000, a
+       dominant override far exceeding the leftover pool's actual
+       range): +19.8%/+4.9%/+2.1% nodes -- worse than the tier escape.
+       A dominant override apparently inherits the escape's downside
+       (jumping ahead of comparably-good moves) without its upside
+       (EFP/pruning exemption).
+     - Proportionate same-tier nudge (2000, scaled to compete with
+       rather than override PSQT+history+continuation): mixed,
+       net flat-to-negative (+20.5%/+2.0%/-0.11%) -- `ecm_hard_quick`
+       went essentially neutral, but `ecm_ringers` (smallest, noisiest
+       suite) got *worse* than every other variant, suggesting it
+       reacts badly to this kind of reordering regardless of magnitude.
+   sn10M depth-reached on the 30-position cross-sectional sample
+   (`/tmp/typhoon/smoke30.ep_`) stayed flat to within noise (14.72-14.83
+   vs. flat-history-only's 14.83) across every variant.
+
+   Conclusion: `SelectBestWithHistory`'s existing PSQT+history+
+   continuation scoring apparently already captures most of what this
+   signal offers; no mechanism tried converts the calibrated FH%
+   advantage into fewer nodes. `mvMovePair` and its evidence-bucket
+   harness were fully reverted (table, update function, nudge/tier
+   code, bucket counters/print block) -- re-derive from git history if
+   revisited with a different approach. The general
+   `MOVE_ORDERING_EVIDENCE_COLLECTION` infrastructure (flag, `MOI_INC`
+   macro, countermove's own bucket harness) was kept -- reusable for
+   calibrating the next candidate (recapture, flee-to-safety revival).
+
+## Done
+
+1. **DONE (kept) -- Flat (non-depth-scaled) history increment.** Replaced
+   `g_HistoryCounters`/`g_ContinuationHistory`'s old `(depth+1)^2`-scaled
+   growth with Crafty's flat/saturating update (`HISTORY_MAX`/
+   `HISTORY_UPDATE_SHIFT`, chess.h; `_Increment`/`_DecrementMoveHistoryCounter`
+   and continuation counterparts, dynamic.c) -- depth-independent on both
+   growth and decay. Measured against a fresh `fbdcb9d` HEAD baseline
+   (`head_reference/`, refreshed same session):
+     - sd10, curated suites: node count -18.9% (`ecm_confident_quick`),
+       -6.6% (`ecm_hard_quick`), +18.9% (`ecm_ringers`, smallest/noisiest --
+       first-move cutoff rate barely moved there, 91.918% -> 91.914%, so
+       likely a re-search artifact not a real regression); first-move
+       cutoff rate up +0.18pp / +0.16pp on the two larger suites.
+     - sn10M, 30-position random sample from `cross_sectional_1.ep_`
+       (seed 42, `/tmp/typhoon/smoke30.ep_`): avg deepest iteration
+       14.52 -> 14.83 ply (+0.31), candidate reached equal-or-deeper depth
+       on 27/29 parsed positions, never shallower by more than 1 ply;
+       first-move cutoff rate 89.724% -> 89.967%.
+   Three independent signals (sd10 node count, sd10/sn10M cutoff rate,
+   sn10M depth-reached) all agree in direction. Kept, not yet committed.
+
+## Already covered / not pursuing
+
+- **History malus (decrement on moves tried before the fail-high move).**
+  Already implemented -- `UpdateDynamicMoveOrdering` (dynamic.c) calls
+  `_DecrementMoveHistoryCounter`/`_DecrementContinuationCounter` for every
+  quiet move considered before the one that beat alpha. Not a gap.
+- Crafty's `movgen.c` itself (pure move generation -- captures, checks,
+  evasions, noncaptures, pin detection) contains no ordering/scoring logic;
+  the real ordering machinery is `next.c` (phase FSM) + `history.c` (killer/
+  countermove/move-pair/history updates).
+
+## Protocol reminder
+
+- One variable at a time, against a fixed reference build (see
+  `head_reference/` convention) -- never "working tree a few hours ago."
+- `sd` (fixed depth), not `sn`/`st`, for "does this shrink the tree"
+  questions -- matches what all five ideas above are actually asking.
+- Read node-count and first-move-cutoff-rate off the three curated suites,
+  not solve-count alone (too noisy at 90-ish positions per CLAUDE.md).
+- `cross_sectional_200k.ep_` at fixed `sn` as a larger tertiary cross-check,
+  not the primary signal, and not on a 100-position sample (too small for
+  an aggregate depth-at-fixed-nodes metric) -- use at least a few thousand
+  positions if run.