Phase A NNUE: feature-indexing spec, annotation/training pipeline, dump nnuefeatures

Builds out the NNUE Phase A prototype per docs/NNUE.md:

- src/nnuefeatures.c: standalone HalfKAv2_hm feature-index reference
  implementation (new `dump nnuefeatures` command), sourced from
  Stockfish's half_ka_v2_hm.{h,cpp} (sf_16 tag), including the 0x88->64
  square conversion typhoon's own board representation needs.
- annotate_pgns/: sharded TWIC/self-play annotation harness
  (annotate.py), position extraction, quiet-position filtering (flat
  and game-shaped, plus an upstream --prefilter-quiet option),
  train/holdout game-level stratified split, WDL-sigmoid K fitting,
  the Python HalfKAv2_hm reference encoder (nnue_feature_spec.py) and
  its bit-for-bit cross-checker against the C side
  (cross_check_features.py), and a first PyTorch training pipeline
  (nnue_model.py, nnue_train.py, build_training_tensors.py) -- a
  256x2-32-32-1-ish net trained locally on CPU against ~1.85M quiet
  TWIC positions as a pipeline/framework validation pass.
- Finish an in-progress InitMagic/InitLMRTable/InitEval ->
  InitializeBitboardMagic/InitializeLMRTable/InitializeEval rename:
  chess.h's prototypes hadn't been updated to match the already-renamed
  definitions/call sites, leaving 4 implicit-function-declaration
  warnings.
- tests/arasan.epd: new curated EPD suite.
- docs/: add the project's design docs (previously untracked).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 254 insertions, 0 deletions

diff --git a/src/docs/ANALYZE_GAME.md b/src/docs/ANALYZE_GAME.md
new file mode 100644
index 0000000..2576a1e
--- /dev/null
+++ b/src/docs/ANALYZE_GAME.md
@@ -0,0 +1,254 @@
+# Analyzing a game from blunders.pgn
+
+A working playbook for pulling a specific game out of `blunders.pgn` and
+figuring out *where the eval/search actually went wrong*, as opposed to
+just noticing that the final result was bad. Written after a session spent
+doing exactly this on the 2026-10-06 `typhoon vs Crafty-25.2` loss (a
+KRP-vs-KQ endgame that looked like it had a scoring bug but turned out to
+be legitimately lost, plus two real engine bugs found along the way).
+
+## 0. What's actually in blunders.pgn
+
+Each game's move comments are **only** `{score/depth time}`, e.g.
+`{-5.09/34 6}` -- score in pawns from White's perspective (not side-to-move;
+confirmed by the sign trending consistently negative as White loses, across
+both White's and Black's own move comments), search depth reached, and
+wall-clock seconds spent on that move. **There is no PV text and no engine
+tag in the PGN at all** -- not `<REC>`, not `<EGTB>`, not `<#>`, nothing.
+Whatever wrapper plays these self-play/vs-Crafty games strips the PV down to
+just the eval number before writing the comment. If you want to see *why*
+a move was chosen -- the actual PV, or whether a recognizer/tablebase hit
+produced the score -- you have to replay the position through a live engine
+yourself; the PGN alone can't tell you.
+
+Don't go looking for `<EGTB>`/`<#>` tags in old PGNs for another reason too:
+those sentinels (`EGTBMOVE` plus the `RECOGN_EGTB` wiring) are new, added in
+the 2026-10-06 session that produced this document. Any game recorded before
+that commit landed couldn't have produced them even if the PGN format did
+carry PV text.
+
+## 1. Pulling the game and exact FENs with python-chess
+
+`python3 -c "import chess"` works in this environment (python-chess is
+installed) -- use it rather than hand-parsing SAN, especially for games deep
+enough that move numbers and ply counts diverge from intuition.
+
+```python
+import chess.pgn
+f = open('blunders.pgn')
+games = []
+g = chess.pgn.read_game(f)
+while g:
+    games.append(g)
+    g = chess.pgn.read_game(f)
+last = games[-1]            # or games[N] for a specific game
+board = last.board()
+moves = list(last.mainline_moves())
+```
+
+**Getting a FEN at a specific PGN move number**: ply count and PGN move
+number are NOT the same thing and it's easy to get this wrong (did exactly
+this once already this session -- cost a wasted round-trip). The mapping:
+
+```python
+def ply(movenum, color):           # color: 'w' or 'b'
+    return 2*(movenum-1) + (1 if color == 'w' else 2)
+```
+
+`moves[ply(69, 'w') - 1]` is White's 69th move (0-indexed list, 1-indexed
+ply). Push moves up through the target ply and read `board.fen()` --
+`board.halfmove_clock` is directly available too and matters a lot (see
+"fifty-move-rule proximity" below).
+
+To scan a whole stretch and watch halfmove_clock / material change move by
+move (useful for finding exactly where a pawn/capture resets the fifty-move
+counter, or where material transitions from one tablebase signature to
+another):
+
+```python
+for i, m in enumerate(moves, start=1):
+    before = board.copy()
+    board.push(m)
+    if lo <= i <= hi:
+        print(i, before.san(m), 'halfmove_clock=', board.halfmove_clock,
+              'fen=', board.fen())
+```
+
+## 2. Replaying a position through typhoon
+
+Use `--batch --command` with `setboard`, not interactive stdin piping (per
+CLAUDE.md, piping commands over stdin to a live REPL doesn't reliably
+complete a multi-step script). Always pass `--egtbpath` explicitly if you
+want to be sure which EGTB directory is in play (the hardcoded default is
+`/zscratch/egtb`, which currently has full WDL+DTZ coverage through 6-man --
+see CLAUDE.md -- but don't assume that silently when writing up findings;
+state it).
+
+```sh
+./typhoon --cpus 1 --hash 256m --egtbpath /zscratch/egtb \
+  --logfile /tmp/typhoon/<name>.log \
+  --batch --command "force; setboard <fen>; eval; sd <depth>; go"
+```
+
+- `eval` alone (no search) gives the raw static eval -- useful as a sanity
+  baseline before trusting a search score.
+- `sd <depth>` fixes search depth; prefer this over `st <seconds>` for
+  reproducibility (machine load varies a lot on this shared box -- see
+  CLAUDE.md's ECM section).
+- Watch stdout for these diagnostic lines, which tell you *how* the move was
+  chosen, not just what it was:
+  - `ONLY MOVE --> stop searching now` -- forced, no real search happened.
+  - `TABLEBASE MOVE --> stop searching now` -- root EGTB (DTZ) root-move
+    short-circuit; score will show as `PV=<EGTB>` and a flat `±NMATE`-ish
+    "mate" display that is **not** a real mate distance (see section 4).
+  - `FORCED MATE --> stop searching now` -- the iterative loop detected a
+    mate score at the current depth and is stopping; the printed PV comes
+    from `szLastPV`, which on a **root fail-high** is just the bare move
+    (`MoveToSan`), not a real PV walk -- see section 5, don't expect `<#>`
+    here.
+  - `DEPTH LIMIT --> stop searching now` -- ordinary completion, nothing
+    special.
+- Grep the output/log for `tellothers` to get the final
+  `score, time, nps, PV=...` summary line in one place.
+
+## 3. Cross-checking with Crafty
+
+`/usr/local/bin/crafty` is available and gives a genuinely independent
+second opinion -- different engine, different (or absent) EGTB usage,
+different search. Use it exactly the way you'd use typhoon:
+
+```sh
+echo "setboard <fen>
+sd <depth>
+go
+quit" | timeout 20 /usr/local/bin/crafty 2>&1 | tail -30
+```
+
+Crafty's output is far chattier (full eval component breakdown, PV with
+move count, nodes/nps) -- look for the `depth ... score ... variation`
+block and the `note: scores are for the white side` line (confirms sign
+convention, which is directly comparable to typhoon's White-POV scores).
+
+**Why this matters**: if typhoon reports something wild (a huge mate-ish
+score with a PV that goes nowhere, or a score frozen for 20+ plies in a row)
+and Crafty reports an ordinary, non-mate score with a PV that's actually
+converging, that's your signal the *typhoon* number is suspect, not that the
+position itself is unusual. This exact comparison is what caught the
+`ProbeEGTBRoot` DTZ-as-mate-distance bug (section 4) -- typhoon said
+`-MATE4`, Crafty's independent 16-ply search said `-7.38` with no mate
+anywhere in a 13-move PV.
+
+## 4. Things that look like bugs but are fifty-move-rule artifacts (and one that was a real bug)
+
+**Watch `halfmove_clock` whenever a long endgame grind produces a
+suspicious score.** Syzygy tables report **WDL** (win/draw/loss) and
+**DTZ** (distance to the next fifty-move-rule-*zeroing* move -- a capture or
+pawn push), not DTM (distance to mate). This project's EGTB directory has
+**zero `.rtbm` files** -- DTM tables were never downloaded, and Fathom's DTM
+code (`probe_dtm_table`, `TB_probe_dtm`, etc.) is present in the source but
+dead weight without data behind it. So:
+
+- A genuinely won endgame can sit at a **small DTZ** while still being many
+  plies from actual mate, especially once `halfmove_clock` is climbing
+  toward 100 (a reset becomes forced soon regardless of how far mate
+  actually is).
+- A score that stays flat for many consecutive moves, or a score that
+  jumps to a huge "mate-ish" magnitude with a PV that doesn't resolve
+  anywhere close to that many plies later, is the signature of this --
+  **not** evidence of missing/corrupt tablebase files by itself. Check
+  `ls /zscratch/egtb | grep <material-signature>` to rule out a genuine
+  coverage gap first, but if the files are present, suspect the
+  DTZ-vs-mate-distance conflation next.
+- This was a **real, fixed bug** as of 2026-10-06: `probe.c`'s
+  `ProbeEGTBRoot` used to compute `*piScore = INFINITY_SCORE ∓ uDtz` and
+  report it through the mate-score display path -- i.e. it fabricated a
+  "mate in N" number out of a quantity that was never a mate distance. Fixed
+  to report a flat `+NMATE`/`-NMATE` (same convention `ProbeEGTB`/
+  `RecognLookup` already use at interior nodes) instead. If you're
+  diagnosing a *pre-fix* game (anything before this fix landed -- check
+  `git log probe.c`), expect to see exactly this artifact and don't chase
+  it as a new finding.
+- The **move** Fathom picks at the root is unaffected by any of this --
+  `tb_probe_root()` does its own internal DTZ-based ranking before
+  returning, so the chosen move is already DTZ-optimal regardless of
+  whether the *score* we derived from it was sane. Don't second-guess the
+  move choice because the displayed score looked wrong; check the score
+  logic and the move logic as separate questions.
+
+## 5. Reading PV sentinels (and when you won't see them)
+
+`WalkPV` (`util.c`) terminates a printed PV early and appends a tag when it
+hits one of five sentinel "moves" (`data.c`): `HASHMOVE` → `<TT>`,
+`RECOGNMOVE` → `<REC>`, `DRAWMOVE` → `<=>`, `MATEMOVE` → `<#>`, `EGTBMOVE` →
+`<EGTB>`. These mark *why the PV stopped*, not what the move was:
+
+- `<REC>` / `<EGTB>`: an interior-node `RecognLookup` hit resolved the
+  score outright. `RECOGN_EGTB` (→ `<EGTB>`) means an actual on-disk Syzygy
+  WDL probe succeeded; `RECOGN_EXACT` (→ `<REC>`) means one of `recogn.c`'s
+  hand-written classic recognizers (KPK, KBKP, etc.) fired instead. Before
+  2026-10-06 these were indistinguishable in the PV (both reported
+  `RECOGNMOVE`) -- `RecognLookup` had an `RECOGN_EGTB` enum value sitting
+  unused the whole time, always returning `RECOGN_EXACT` even on a genuine
+  Fathom hit. If you're looking at an old log/build, `<REC>` could mean
+  either; `<EGTB>` specifically didn't exist yet.
+- `<#>`: search found an actual forced checkmate (zero legal moves + in
+  check) at a PV leaf.
+- `<=>`: a drawn leaf (zero legal moves, not in check == stalemate; this
+  sentinel is also used for recognized/forced draws elsewhere).
+- **Root tablebase moves get `<EGTB>` in `szLastPV` directly** (as a literal
+  string, not via `WalkPV` -- `ProbeEGTBRoot`'s success path short-circuits
+  the whole iterative loop and never calls `WalkPV` at all for that move).
+
+**The big gotcha**: `PrintPV` (`util.c`) only calls the real `WalkPV` walk
+-- the one that can surface any of these tags -- when the reported score is
+a genuine PV node, `iAlpha < iScore < iBeta`. On a **root fail-high**
+(`iScore > iBeta`, i.e. the aspiration window is still widening upward),
+it falls back to printing just the bare best move via `MoveToSan`, with no
+PV walk and therefore no sentinel, ever -- even if the position is a
+dead-obvious mate-in-1 whose leaf genuinely wrote `MATEMOVE` two plies down.
+So: **don't read "no `<#>` in the output" as "no mate was found"** -- check
+whether the line says `FORCED MATE` (a fail-high-driven stop) vs. a normal
+depth-complete report before concluding anything about whether the engine
+actually resolved the mating line as an exact PV.
+
+## 6. The timer-elapsed-time gotcha
+
+If you see `Searched for -<huge number> seconds` in output, that's not a
+real timing anomaly to chase -- or rather, it *was* a real bug (fixed
+2026-10-06): `Iterate()` (`root.c`) used to stamp
+`g_MoveTimer.dEndTime = SystemTimeStamp()` partway through the function,
+*before* the shared `end:` label that every early-exit path (`ONLY MOVE`,
+`TABLEBASE MOVE`, pre-move terminal states) jumps to via `goto end`. Those
+paths skipped the stamp entirely, leaving `dEndTime` stale (0, or whatever
+the previous unrelated search last set it to), which produces exactly this
+kind of nonsense negative "elapsed time" when the end-of-search report
+computes `dEndTime - dStartTime`. Fixed by moving the stamp to right after
+the `end:` label itself. If you're diagnosing a pre-fix build and see this,
+it's cosmetic (doesn't affect move choice or score), but it's also a tell
+that you're looking at a root short-circuit path (tablebase or only-move),
+which is useful context for *why* no real iterative search happened at that
+node.
+
+## 7. General workflow summary
+
+1. Identify the game (usually "last game in the file") and get the exact
+   FEN at the move(s) you care about via python-chess -- don't eyeball SAN
+   move numbers against ply counts by hand.
+2. Check `board.halfmove_clock` at that FEN before anything else if it's an
+   endgame -- rules out/in the fifty-move-rule class of artifact immediately.
+3. Replay the FEN through `./typhoon --batch --command "force; setboard
+   <fen>; sd <depth>; go"`, read the diagnostic `--> stop searching now`
+   line to understand *which* code path produced the move, and check
+   `tellothers`'s `PV=` field.
+4. Get a second opinion from `/usr/local/bin/crafty` on the same FEN at
+   comparable depth. Large disagreement (especially "my engine claims near-
+   mate, Crafty's PV goes nowhere near that far") is the signal to dig into
+   the score-computation code, not the position.
+5. If something still looks wrong, read the actual source for the code path
+   the diagnostic line pointed at (`root.c`'s `Iterate`, `probe.c`'s
+   `ProbeEGTB`/`ProbeEGTBRoot`, `recogn.c`'s `RecognLookup`) rather than
+   guessing from behavior alone -- every bug found this way so far
+   (`ProbeEGTBRoot`'s DTZ-as-mate-score, the `dEndTime` placement, the
+   dead `RECOGN_EGTB` branch) was a straightforward, small, findable-in-
+   minutes fix once the actual code was in front of us; none of them
+   required deep speculation.