NNUE search integration: FH% tables, NODE_TYPE refinement, TREEDUMP rewrite

- data.c/chess.h: widen the joint FH% table's late axis 16->17, dedicating
  the new bucket 0 to a whole-node ("will something here fail high at
  all") sample rather than sharing it with the first-leftover-move curve.
  Add an NNUE-eval counterpart table (g_uJointFHThousandths_NNUE) and an
  NNUE counterpart of the beta-side table, selected at runtime based on
  whether NNUE eval is active.
- search.c/searchsup.c: add _RefineNodeTypeGuess, which uses the FH%
  tables (via g_NodeTypeGuessTable) to refine a node's NODE_TYPE guess
  instead of the old heuristic; thread uParentMoveFHEstimate through
  PLY_INFO so a node can see its parent's prediction for the move that
  led to it.
- generate.c: initialize sKillers[2]/[3].uBonus unconditionally (an
  uninitialized-stack read caught by valgrind under LTO; harmless today
  but a real bug).
- dumptree.c/chess.h/GNUmakefile: replace the long-dead DUMP_TREE XML
  tree dump (written for a GUI this project no longer has) with TREEDUMP,
  a flat, path-keyed move/result trace meant to diff against an
  identically-instrumented Stockfish build -- see docs/TREEDUMP_FORMAT.md.
- docs/DEEPER_PRUNING.md: notes on closing the search-efficiency gap vs
  Stockfish.
- crafty_egtb/stockfish_xboard/typhoon_nnue: wrapper scripts so
  match_play.py's Engine class can drive Crafty, Stockfish (via
  PolyGlot), and an NNUE-enabled typhoon the same way it drives HCE
  typhoon.
- tests/lop: rename lop.txt -> lop.ep_; update compare_lop.cpp.
- annotate_pgns: add aggregate_marginsafety.py, dedup_pgn.py,
  fit_lazy_probe.py; update annotate.py/run_nnue_pipeline.py for the NNUE
  training pipeline.
- .gitignore: ignore compile_commands.json (clangd build DB) and
  eval.nnue (a symlink into the untracked head_reference/ comparison
  baseline), neither of which belongs in the repo.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 64 insertions, 0 deletions

diff --git a/src/stockfish_xboard b/src/stockfish_xboard
new file mode 100755
index 0000000..91765ad
--- /dev/null
+++ b/src/stockfish_xboard
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+#
+# stockfish_xboard -- thin wrapper so match_play.py's Engine class (which
+# always spawns its path with hardcoded TYPHOON flags, "--hash 256m --cpus 1
+# --logfile -" -- see match_play.py's engine_args) can drive Stockfish
+# instead. Stockfish speaks UCI, not xboard, so this execs it through
+# PolyGlot (the UCI<->xboard adapter already installed on this box,
+# /usr/local/bin/polyglot) and deliberately DISCARDS every argument it's
+# called with -- match_play.py's hardcoded typhoon-style flags are
+# meaningless (and would error) if passed to polyglot/Stockfish directly.
+#
+# Same "ignore whatever the caller passes, always do the right thing"
+# pattern as the NNUE_WEIGHTS shim in eval_tune/test_vs_head.sh.
+#
+# Verified manually (2026-10-17): PolyGlot's xboard handshake (feature
+# negotiation ending "feature done=1") and a real "setboard ...; st 2; go"
+# round-trip both work correctly when driven by a persistent subprocess
+# that keeps stdin open (exactly how Engine class drives it) -- a one-shot
+# shell pipe (`printf ... | polyglot`) does NOT work, stdin EOFs before
+# Stockfish finishes thinking and no "move ..." line is ever printed. Do
+# not "simplify" this to a plain pipe when debugging.
+#
+# Override the Stockfish binary/dir via env vars if needed:
+#   STOCKFISH_BIN   defaults to "stockfish" (resolved via PATH, i.e.
+#                   /usr/local/bin/stockfish on this box)
+#   STOCKFISH_DIR   defaults to /usr/local/bin (polyglot's -ed)
+#   SYZYGY_PATH     defaults to /zscratch/egtb (same dir typhoon's own
+#                   --egtbpath default points at, see main.c). Stockfish 19's
+#                   tbprobe.cpp is itself Fathom-derived, so it reads the
+#                   exact same .rtbw/.rtbz files typhoon does -- no separate
+#                   Stockfish-format tablebase or conversion needed.
+#                   Verified 2026-10-18: `setoption name SyzygyPath value
+#                   /zscratch/egtb` against a bare stockfish UCI session logs
+#                   "info string Found 510 WDL and 510 DTZ tablebase files
+#                   (up to 6-man)." PolyGlot forwards this through via its
+#                   `-uci SyzygyPath=...` flag (any `-uci name=value` is
+#                   passed straight to the engine as a UCI setoption at
+#                   startup, before the xboard-side handshake). Set to ""
+#                   to disable TB probing entirely.
+SYZYGY_PATH="${SYZYGY_PATH:-/zscratch/egtb}"
+#
+# 2026-10-17: wrapped with `stdbuf -oL` -- when PolyGlot's stdout isn't
+# a tty (true here: match_play.py spawns it via subprocess.Popen with a
+# pipe), C stdio defaults to full block buffering instead of line
+# buffering, so xboard protocol lines (including "move ...") can sit
+# unflushed in PolyGlot's own buffer while match_play.py's readline()
+# blocks waiting for them -- looks exactly like an I/O-bound engine
+# from the outside (low measured CPU, long wall-clock per move, idle
+# machine) even though the actual Stockfish search finished promptly.
+# Same class of bug as this session's own repeated Python stdout-
+# buffering gotchas, just at the C-stdio layer instead. `stdbuf -oL`
+# forces line buffering regardless of tty status.
+
+set -e
+
+STOCKFISH_BIN="${STOCKFISH_BIN:-stockfish}"
+STOCKFISH_DIR="${STOCKFISH_DIR:-/usr/local/bin}"
+
+if [ -n "$SYZYGY_PATH" ]; then
+    exec stdbuf -oL polyglot -noini -ec "$STOCKFISH_BIN" -ed "$STOCKFISH_DIR" \
+        -uci "SyzygyPath=$SYZYGY_PATH"
+else
+    exec stdbuf -oL polyglot -noini -ec "$STOCKFISH_BIN" -ed "$STOCKFISH_DIR"
+fi