NNUE search integration: FH% tables, NODE_TYPE refinement, TREEDUMP rewrite

- data.c/chess.h: widen the joint FH% table's late axis 16->17, dedicating
  the new bucket 0 to a whole-node ("will something here fail high at
  all") sample rather than sharing it with the first-leftover-move curve.
  Add an NNUE-eval counterpart table (g_uJointFHThousandths_NNUE) and an
  NNUE counterpart of the beta-side table, selected at runtime based on
  whether NNUE eval is active.
- search.c/searchsup.c: add _RefineNodeTypeGuess, which uses the FH%
  tables (via g_NodeTypeGuessTable) to refine a node's NODE_TYPE guess
  instead of the old heuristic; thread uParentMoveFHEstimate through
  PLY_INFO so a node can see its parent's prediction for the move that
  led to it.
- generate.c: initialize sKillers[2]/[3].uBonus unconditionally (an
  uninitialized-stack read caught by valgrind under LTO; harmless today
  but a real bug).
- dumptree.c/chess.h/GNUmakefile: replace the long-dead DUMP_TREE XML
  tree dump (written for a GUI this project no longer has) with TREEDUMP,
  a flat, path-keyed move/result trace meant to diff against an
  identically-instrumented Stockfish build -- see docs/TREEDUMP_FORMAT.md.
- docs/DEEPER_PRUNING.md: notes on closing the search-efficiency gap vs
  Stockfish.
- crafty_egtb/stockfish_xboard/typhoon_nnue: wrapper scripts so
  match_play.py's Engine class can drive Crafty, Stockfish (via
  PolyGlot), and an NNUE-enabled typhoon the same way it drives HCE
  typhoon.
- tests/lop: rename lop.txt -> lop.ep_; update compare_lop.cpp.
- annotate_pgns: add aggregate_marginsafety.py, dedup_pgn.py,
  fit_lazy_probe.py; update annotate.py/run_nnue_pipeline.py for the NNUE
  training pipeline.
- .gitignore: ignore compile_commands.json (clangd build DB) and
  eval.nnue (a symlink into the untracked head_reference/ comparison
  baseline), neither of which belongs in the repo.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 191 insertions, 0 deletions

diff --git a/src/docs/TREEDUMP_FORMAT.md b/src/docs/TREEDUMP_FORMAT.md
new file mode 100644
index 0000000..ecc4325
--- /dev/null
+++ b/src/docs/TREEDUMP_FORMAT.md
@@ -0,0 +1,191 @@
+# TREEDUMP: a shared search-tree trace format for typhoon vs. Stockfish
+
+Both engines, when built with a `TREEDUMP` compile-time flag, write a
+flat, line-oriented, always-on trace of search decisions to a file
+named `treedump.log` in the current directory. There is no runtime
+toggle or filtering -- keep a dump small by using a shallow `sd N`
+(typhoon) / `go depth N` (Stockfish), not by restricting what gets
+written.
+
+The point is to run both engines on the *same FEN* at the *same shallow
+depth*, then diff/meld the two `treedump.log` files to see exactly
+where their trees diverge -- which move each engine reduces, prunes, or
+searches in full, and by how much.
+
+## Building
+
+- typhoon: `gmake release PERF_COUNTERS=1 TREEDUMP=1` (or `debug`).
+  Writes `treedump.log` in the working directory from process start to
+  exit (`TDOpen`/`TDClose` in `main.c`). Single-threaded runs only
+  (`--cpus 1`) -- the file write isn't locked against concurrent
+  threads, matching this project's own `--cpus 1` convention for
+  diagnostic/comparison runs.
+- Stockfish: `gmake build ARCH=x86-64-avx2 COMP=clang EXTRACXXFLAGS=-DTREEDUMP`
+  (FreeBSD: use `gmake`, not `make` -- the stock `make` chokes on this
+  Makefile's GNU-only syntax; `COMP=clang` because there's no `gcc` on
+  this box; `ARCH=x86-64-avx2`, not bare `x86-64` or `ARCH=native` --
+  both of the latter either fail FreeBSD's arch-detection outright or
+  silently build a popcnt-less binary that reproducibly returns a
+  dead/empty search, 0 nodes, on every `go`, with no error -- see the
+  gotcha below, this is a different failure mode from it and easy to
+  conflate with it). Also writes `treedump.log` in the working
+  directory, for the life of the process. Single-threaded only
+  (`Threads` UCI option defaults to 1; don't raise it for a comparison
+  run). **`EXTRACXXFLAGS` isn't tracked as a `make` dependency** -- if
+  you rebuild the same ARCH with/without it back to back, `search.cpp`
+  (the only file TREEDUMP touches) won't be recompiled and you'll
+  silently get the previous binary; `touch search.cpp` first, or
+  `gmake clean` first, whenever toggling the flag.
+
+### A Stockfish-specific gotcha: piping input races the search thread
+
+If you drive Stockfish non-interactively by piping commands in and
+letting the pipe close right after (`printf '...\ngo depth N\n' |
+./stockfish`), the engine's input-reading thread can see EOF and begin
+shutting the engine down *before* the search thread has actually
+started producing nodes -- the symptom is `info depth 1 ... nodes 0
+... pv ` (empty) immediately followed by a `bestmove`, for every
+UCI limit type (`depth`, `nodes`, `movetime` all equally ignored), with
+exit code 0 (not a crash). This reproduced on this box on a **clean,
+unmodified** Stockfish build too, so it's not a TREEDUMP artifact --
+it's inherent to how this fork's shared-memory NNUE bootstrap
+(`shm.h`/`shm_unix.h`, "Network replica N: Shared memory" in the
+startup banner) races against stdin closing, and it was intermittent
+enough (roughly 1 good run in ~15-18 attempts, observed empirically)
+that blind retrying is not a reliable workaround.
+
+**Fix: hold stdin open past the point the search should finish**,
+e.g.:
+
+```sh
+(printf 'position fen <FEN>\ngo depth N\n'; sleep 2) | ./stockfish
+```
+
+An interactive terminal session (typing the commands directly, or
+redirecting from a real tty) never hits this, which is why it's easy
+to not reproduce it while debugging interactively and then hit it
+again the next time a script pipes input in.
+
+## Why path-keyed, not node-numbered
+
+Internal node counters are meaningless across two different engines --
+their recursion order, node numbering, and even which positions get
+revisited (transposition table hits) differ completely. The only thing
+that identifies "the same position" in both engines is the sequence of
+moves played from the root, since both start from the same FEN. Every
+line is therefore keyed by `path="<uci moves from root, space-separated>"`
+-- sort or `grep` on that field to line up corresponding decisions from
+the two logs regardless of visit order.
+
+A null move is spelled `0000` in a path (both engines already use this
+UCI convention for `Move::null()`/`NULLMOVE` internally -- it's not an
+invented token).
+
+## Record types
+
+Two kinds of line, both single-line, both flushed immediately (so a
+killed/crashed process still leaves a readable partial log):
+
+```
+MOVE path="<path, ending in the move this line is about>" depth=<float> alpha=<int> beta=<int> kind=<str> act=<str> r=<float> ext=<float> score=<int|SKIPPED>
+
+RESULT path="<path, NOT including any move at this node>" depth=<float> alpha=<int> beta=<int> eval=<int> bound=<EXACT|LOWER|UPPER> score=<int>
+```
+
+- `depth`: remaining search depth in whole plies (float, since
+  typhoon's internal depth is fixed-point in `ONE_PLY`-sized ticks and
+  extensions/reductions can be fractional ply amounts; Stockfish's is
+  already integer plies and just prints with no fractional part).
+- `alpha`/`beta`: the window at the node the move/result belongs to.
+  For a `RESULT` line, this is the node's *original* (pre-aspiration-
+  widening) window, so the `bound` field's classification is
+  consistent.
+- `kind` (MOVE only): rough move classification -- `FIRST`, `HASH`,
+  `CAP`, `QUIET`, `NULL`, `PROBCUT`, freeform otherwise. Not identical
+  granularity between the two engines (see Known Gaps below) --
+  treat it as a hint, not a guaranteed-matching taxonomy.
+- `act` (MOVE only): what happened to this move --
+  `FULL` (searched at full/nominal depth), `LMR` (late-move-reduced),
+  `LMR-RESEARCH` (Stockfish only: a reduced search failed high and got
+  a full-depth re-search), `EXT` (extended), `LMP` (pruned without
+  being searched, "late move pruning"), `FP-CAP`/`SEE-CAP`/
+  `HIST-PRUNE`/`FP-QUIET`/`SEE-QUIET` (Stockfish's shallow-depth move
+  pruning variants -- typhoon currently has no direct equivalent to
+  several of these, which is itself one of the interesting things to
+  look for in a diff), `NMP` (null-move probe), `PROBCUT` (Stockfish
+  only -- typhoon has no ProbCut at all).
+- `r`/`ext` (MOVE only): magnitude, in whole plies, of any reduction
+  (`r`) or extension (`ext`) applied. Exactly one is normally nonzero.
+- `score`: the score this move/node returned, or the literal `SKIPPED`
+  on a `MOVE` line for a move that was pruned without ever being
+  searched.
+- `eval` (RESULT only): the node's static eval. May be stale/0 on a
+  typhoon node that returned before `Eval()` ran (e.g. an immediate
+  hash/recognizer cutoff) -- diagnostic only, don't trust it blindly.
+
+## Known gaps / asymmetries (read before drawing conclusions from a diff)
+
+- **Neither engine traces into qsearch.** Qsearch nodes vastly
+  outnumber full-width nodes even at `sd 4`/`go depth 4` and would
+  swamp a diff of the full-width move-ordering/pruning decisions that
+  are actually of interest. This means a TREEDUMP diff only shows the
+  full-width tree shape, not what happens once captures start
+  resolving near the horizon -- keep comparisons shallow enough that
+  qsearch isn't where most of the interesting divergence lives, or
+  treat this as future work if it turns out qsearch matters too.
+- **`kind` is not a matched taxonomy.** typhoon's `TDMove` call sites
+  classify moves coarsely (`FIRST`/`CAP`/`QUIET`/`NULL`); Stockfish's
+  do the same but the two engines' internal move-ordering concepts
+  (killers, countermoves, TT move, etc.) don't line up 1:1. Don't
+  diff on `kind` expecting an exact match -- diff on `act`, `r`/`ext`,
+  and `score` for a given `path`, and use `kind` only as a sanity
+  check.
+- **typhoon has no ProbCut.** Any `act=PROBCUT` line will only ever
+  appear in Stockfish's log. This is itself a structural difference
+  worth noting, not a bug in the instrumentation.
+- **Stockfish's null-move `depth` field is the full pre-reduction
+  parent depth minus R; typhoon's null-move `MOVE` line's `depth`
+  field is the already-reduced probe depth** (`TryNullmovePruning`
+  never receives the pre-reduction parent depth) -- compare the
+  *absolute* depth value in context, don't assume the two `r` fields
+  for null moves are computed the same way (typhoon's is left `0.00`
+  for null moves rather than faked).
+- **typhoon's `RESULT.eval` can be stale (0 or a leftover value)** on
+  any node that returned before `Eval()` ran. Stockfish's `eval` field
+  comes from `ss->staticEval`, which has its own, different set of
+  early-return paths that skip a real eval. Don't diff raw eval
+  numbers across engines expecting them to be comparable at all (they
+  use different evaluation functions) -- only use `eval` to sanity-
+  check a single engine's own `RESULT` line against its own earlier
+  `MOVE` lines.
+
+## Suggested workflow
+
+```sh
+# typhoon side
+cd /usr/home/scott/typhoon/src
+gmake release PERF_COUNTERS=1 TREEDUMP=1
+rm -f treedump.log
+./typhoon --cpus 1 --hash 8m --logfile - --batch --command \
+    "force; book name /nonexistent.book.bin; setboard <FEN>; sd 4; go"
+cp treedump.log /tmp/typhoon/td_typhoon.log
+
+# stockfish side
+cd ~/programming/chess/stockfish/src
+touch search.cpp  # EXTRACXXFLAGS isn't a tracked dependency -- force the recompile
+gmake build ARCH=x86-64-avx2 COMP=clang EXTRACXXFLAGS=-DTREEDUMP
+rm -f treedump.log
+(printf 'position fen <FEN>\ngo depth 4\n'; sleep 2) | ./stockfish
+cp treedump.log /tmp/typhoon/td_sf.log
+
+# compare (sort by path first so corresponding positions line up
+# regardless of each engine's own visit order)
+sort -t'"' -k2 /tmp/typhoon/td_typhoon.log > /tmp/typhoon/td_typhoon.sorted
+sort -t'"' -k2 /tmp/typhoon/td_sf.log > /tmp/typhoon/td_sf.sorted
+meld /tmp/typhoon/td_typhoon.sorted /tmp/typhoon/td_sf.sorted
+```
+
+Rebuild each engine back to its normal release profile afterward
+(`TREEDUMP` should never be the profile left in place for a real run --
+it's diagnostic-only, same spirit as this project's other collection-
+build flags).