NNUE search integration: FH% tables, NODE_TYPE refinement, TREEDUMP rewrite

- data.c/chess.h: widen the joint FH% table's late axis 16->17, dedicating
  the new bucket 0 to a whole-node ("will something here fail high at
  all") sample rather than sharing it with the first-leftover-move curve.
  Add an NNUE-eval counterpart table (g_uJointFHThousandths_NNUE) and an
  NNUE counterpart of the beta-side table, selected at runtime based on
  whether NNUE eval is active.
- search.c/searchsup.c: add _RefineNodeTypeGuess, which uses the FH%
  tables (via g_NodeTypeGuessTable) to refine a node's NODE_TYPE guess
  instead of the old heuristic; thread uParentMoveFHEstimate through
  PLY_INFO so a node can see its parent's prediction for the move that
  led to it.
- generate.c: initialize sKillers[2]/[3].uBonus unconditionally (an
  uninitialized-stack read caught by valgrind under LTO; harmless today
  but a real bug).
- dumptree.c/chess.h/GNUmakefile: replace the long-dead DUMP_TREE XML
  tree dump (written for a GUI this project no longer has) with TREEDUMP,
  a flat, path-keyed move/result trace meant to diff against an
  identically-instrumented Stockfish build -- see docs/TREEDUMP_FORMAT.md.
- docs/DEEPER_PRUNING.md: notes on closing the search-efficiency gap vs
  Stockfish.
- crafty_egtb/stockfish_xboard/typhoon_nnue: wrapper scripts so
  match_play.py's Engine class can drive Crafty, Stockfish (via
  PolyGlot), and an NNUE-enabled typhoon the same way it drives HCE
  typhoon.
- tests/lop: rename lop.txt -> lop.ep_; update compare_lop.cpp.
- annotate_pgns: add aggregate_marginsafety.py, dedup_pgn.py,
  fit_lazy_probe.py; update annotate.py/run_nnue_pipeline.py for the NNUE
  training pipeline.
- .gitignore: ignore compile_commands.json (clangd build DB) and
  eval.nnue (a symlink into the untracked head_reference/ comparison
  baseline), neither of which belongs in the repo.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 31 insertions, 5 deletions

diff --git a/annotate_pgns/run_nnue_pipeline.py b/annotate_pgns/run_nnue_pipeline.py
index 6e51416..0210c53 100644
--- a/annotate_pgns/run_nnue_pipeline.py
+++ b/annotate_pgns/run_nnue_pipeline.py
@@ -20,10 +20,19 @@ specifically to handle a foreign-engine/Stockfish-sourced corpus).
 Chains the existing, already-individually-tested pipeline scripts in
 this order:
 
-    1. extract_selfplay_positions.py  each input PGN -> games.jsonl
-                                       (one run per file, since that
-                                       script takes a single PGN;
-                                       outputs concatenated)
+    0. dedup_pgn.py                   drops duplicate games (same
+                                       starting FEN + move text,
+                                       cross-file) before anything
+                                       else sees them -- self-play's
+                                       randomized starting positions
+                                       can recur, and a recurrence
+                                       that also plays out identically
+                                       is a real duplicate, not just a
+                                       repeated opening
+    1. extract_selfplay_positions.py  each deduped input PGN ->
+                                       games.jsonl (one run per file,
+                                       since that script takes a
+                                       single PGN; outputs concatenated)
     2. split_holdout.py               games.jsonl -> train/holdout
                                        split (by game, not position)
     3. fit_k_and_label.py             fits the WDL-sigmoid K on
@@ -94,7 +103,7 @@ def split_pgn(path, n, outdir):
 
     n_games = len(offsets)
     if n_games == 0:
-        return [path]
+        return [Path(path)]
     n = min(n, n_games)
     boundaries = [offsets[round(i * n_games / n)] for i in range(n)] + [end]
 
@@ -245,6 +254,23 @@ def main():
     rundir = Path(args.rundir)
     rundir.mkdir(parents=True, exist_ok=True)
 
+    # 0. Dedup games across all input files before anything else sees
+    # them. Self-play starting positions are randomized but drawn from
+    # a finite pool, so the same random opening -- and, on an unlucky
+    # draw, the same resulting game -- can recur within or across input
+    # files (see dedup_pgn.py's own docstring). Keyed on (starting FEN,
+    # move text), cross-file (one shared in-memory key set across all
+    # inputs in this single invocation), so a duplicate occurring in a
+    # later input file against an earlier one is still caught.
+    dedup_dir = rundir / "dedup"
+    deduped_paths = [dedup_dir / f"{Path(p).stem}.dedup.pgn" for p in pgn_paths]
+    run_stage(
+        "dedup_pgn.py", deduped_paths,
+        [sys.executable, HERE / "dedup_pgn.py",
+         *pgn_paths, "--out-dir", dedup_dir],
+        args.force)
+    pgn_paths = [str(p) for p in deduped_paths]
+
     games_path = rundir / "games.jsonl"
     train_path = rundir / "games.train.jsonl"
     holdout_path = rundir / "games.holdout.jsonl"