NNUE search integration: FH% tables, NODE_TYPE refinement, TREEDUMP rewrite

- data.c/chess.h: widen the joint FH% table's late axis 16->17, dedicating
  the new bucket 0 to a whole-node ("will something here fail high at
  all") sample rather than sharing it with the first-leftover-move curve.
  Add an NNUE-eval counterpart table (g_uJointFHThousandths_NNUE) and an
  NNUE counterpart of the beta-side table, selected at runtime based on
  whether NNUE eval is active.
- search.c/searchsup.c: add _RefineNodeTypeGuess, which uses the FH%
  tables (via g_NodeTypeGuessTable) to refine a node's NODE_TYPE guess
  instead of the old heuristic; thread uParentMoveFHEstimate through
  PLY_INFO so a node can see its parent's prediction for the move that
  led to it.
- generate.c: initialize sKillers[2]/[3].uBonus unconditionally (an
  uninitialized-stack read caught by valgrind under LTO; harmless today
  but a real bug).
- dumptree.c/chess.h/GNUmakefile: replace the long-dead DUMP_TREE XML
  tree dump (written for a GUI this project no longer has) with TREEDUMP,
  a flat, path-keyed move/result trace meant to diff against an
  identically-instrumented Stockfish build -- see docs/TREEDUMP_FORMAT.md.
- docs/DEEPER_PRUNING.md: notes on closing the search-efficiency gap vs
  Stockfish.
- crafty_egtb/stockfish_xboard/typhoon_nnue: wrapper scripts so
  match_play.py's Engine class can drive Crafty, Stockfish (via
  PolyGlot), and an NNUE-enabled typhoon the same way it drives HCE
  typhoon.
- tests/lop: rename lop.txt -> lop.ep_; update compare_lop.cpp.
- annotate_pgns: add aggregate_marginsafety.py, dedup_pgn.py,
  fit_lazy_probe.py; update annotate.py/run_nnue_pipeline.py for the NNUE
  training pipeline.
- .gitignore: ignore compile_commands.json (clangd build DB) and
  eval.nnue (a symlink into the untracked head_reference/ comparison
  baseline), neither of which belongs in the repo.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 59 insertions, 23 deletions

diff --git a/src/TODO b/src/TODO
index 0d2251b..aac2082 100644
--- a/src/TODO
+++ b/src/TODO
@@ -1,5 +1,3 @@
-Better understanding of stuff like KRKRB == drawish, Q's come off = !drawish
-
 Pawnhash size tunable from cmdline again
 
 Think about when to avoid nullmove pruning based on checks in the line
@@ -19,18 +17,6 @@ it collides with self-play match runs -- schedule it for whenever the
 box isn't already saturated by match_play.py workers, not concurrently
 with them.
 
-2026-09-18: blunders.pgn review (typhoon vs Crafty losses).
-
-2026-09-18: consider adding cLikelyUnstoppablePasser to the POSITION
-structure and using it to extend searches.  Just like we already do
-with cTrapped and cEnprise -- signals from eval.c to affect search.
-
-2026-10-01: It seems like a Q vs RR in a pawnful endgame... Q dominates.
-In general, we should work on encouraging the engine to maintain a good
-pawn structure -- read more of that pawn power in chess book.  Work on
-the pawn eval specifically.  Maybe do eval learning on only the pawn
-hashed eval terms.
-
 2026-10-04: We need to go faster.
  - Can we get rid of rgSquares[] completely and use bitboards everywhere?
  - Eliminate SEE calls?
@@ -60,20 +46,70 @@ hashed eval terms.
 move in the PGN output.  Make self-play harness able to play crafty
 and then test the HEAD/candidate against crafty.
 
-Eval term for connected passers vs. rooks.  Or 2+ passers vs. rooks.
+Remove all line-ending whitespace in one commit.
 
-Poor king play -- tell it to support own passers or blockade enemy
-passers once king activity phase starts.
+Fix the input queue --batch mode one thing after the script; like this breaks it.
 
-Self-play lite and heavy.
+Add nullmoves skipped to the output.  Redo the output in general.
 
-Valgrind in debug_smoke_test.sh.
+Train the fuck out of the NNUE.
 
-Remove all line-ending whitespace in one commit.
+Figure out how Stockfish is so efficient with the tree it searches:
 
-Fix the input queue --batch mode one thing after the script; like this breaks it.
+position fen 2q1r1k1/1ppb4/r2p1Pp1/p4n1p/2P1n3/5NPP/PP3Q1K/2BRRB2 w - -
+go depth 18
+info string Available processors: 0-23
+info string Using 1 thread
+info string NNUE evaluation using nn-1a298aa575a0.nnue (109MiB, (86896, 1024, 32, 32, 1))
+info string Network replica 1: Shared memory.
+info depth 1 seldepth 5 multipv 1 score cp 433 nodes 47 nps 47000 hashfull 0 tbhits 0 time 1 pv e1e4
+info depth 2 seldepth 3 multipv 1 score cp 433 nodes 92 nps 92000 hashfull 0 tbhits 0 time 1 pv e1e4 e8e4
+info depth 3 seldepth 4 multipv 1 score cp 433 nodes 140 nps 140000 hashfull 0 tbhits 0 time 1 pv e1e4 e8e4 f3g5
+info depth 4 seldepth 6 multipv 1 score cp 433 nodes 199 nps 199000 hashfull 0 tbhits 0 time 1 pv e1e4 e8e4 f3g5 e4e6
+info depth 5 seldepth 11 multipv 1 score cp 433 nodes 273 nps 273000 hashfull 0 tbhits 0 time 1 pv e1e4 e8e4 f3g5 e4e6 f6f7 g8g7
+info depth 6 seldepth 11 multipv 1 score cp 433 nodes 347 nps 347000 hashfull 0 tbhits 0 time 1 pv e1e4 e8e4 f3g5 e4e6 f6f7 g8g7
+info depth 7 seldepth 11 multipv 1 score cp 664 nodes 613 nps 613000 hashfull 0 tbhits 0 time 1 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7e7 g5e4
+info depth 8 seldepth 12 multipv 1 score cp 808 nodes 1070 nps 356666 hashfull 0 tbhits 0 time 3 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7f8
+info depth 9 seldepth 11 multipv 1 score cp 762 nodes 1201 nps 400333 hashfull 0 tbhits 0 time 3 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7f8 g5e4 f8g7 e4g5
+info depth 10 seldepth 11 multipv 1 score cp 747 nodes 1387 nps 346750 hashfull 0 tbhits 0 time 4 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7g8 g5e4
+info depth 11 seldepth 12 multipv 1 score cp 707 nodes 1527 nps 381750 hashfull 0 tbhits 0 time 4 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7g8 g5e4 a5a4 e4g5 a6a5 a2a3
+info depth 12 seldepth 15 multipv 1 score cp 630 nodes 1864 nps 310666 hashfull 0 tbhits 0 time 6 pv f6f7 g8f7 e1e4 d7a4 f3g5 f7g8 e4e8 c8e8 d1d5
+info depth 13 seldepth 15 multipv 1 score cp 725 nodes 2579 nps 286555 hashfull 1 tbhits 0 time 9 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7f8 g5e4 f8g8 d1e1
+info depth 14 seldepth 17 multipv 1 score cp 659 nodes 3095 nps 281363 hashfull 1 tbhits 0 time 11 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7g8 g5e4 d7c6 e4g5 a5a4 g3g4 h5g4
+info depth 15 seldepth 21 multipv 1 score cp 713 nodes 8091 nps 212921 hashfull 3 tbhits 0 time 38 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7g8 g5e4 h5h4 g3h4 d7a4 d1e1 a4c6 e4g5 c6a4 e1e6
+info depth 16 seldepth 21 multipv 1 score cp 776 nodes 12732 nps 205354 hashfull 4 tbhits 0 time 62 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7g8 g5e4 d7c6 e4g5 c8f8 f1g2 a5a4 g2d5 c6d5 c4d5
+info depth 17 seldepth 20 multipv 1 score cp 766 nodes 16215 nps 202687 hashfull 6 tbhits 0 time 80 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7g8 g5e4 d7c6 e4g5 c8f8 c4c5
+info depth 18 seldepth 20 multipv 1 score cp 745 nodes 20330 nps 209587 hashfull 7 tbhits 0 time 97 pv f6f7 g8f7 e1e4 e8e4 f3g5 f7g7 g5e4 c8e8 d1e1 e8e5 e4g5 e5c5 f2c5 d6c5 g3g4 h5g4 h3g4 f5h6
+bestmove f6f7 ponder g8f7
 
-Add nullmoves skipped to the output.  Redo the output in general.
+vs
+
+The root position is: 2q1r1k1/1ppb4/r2p1Pp1/p4n1p/2P1n3/5NPP/PP3Q1K/2BRRB2 w - - 0 0
+ 1  -0.03  00:00.000  11           Kg1 Nxf6
+ 1  +0.02  00:00.000  47           Qc2 Nfxg3
+ 1  +0.75  00:00.000  174          f7+ Kxf7 Ng5+ Nxg5 Rxe8
+ 2  +0.02  00:00.001  242          Qc2 Nfxg3
+ 3  +0.12  00:00.003  985          Qc2 Nexg3 Kg1 Rxe1
+ 3+ +0.77  00:00.004  1173         Qc2 Nexg3 Kg1 Rxe1  ++
+ 3+ +2.27  00:00.004  1201         Qc2 Nexg3 Kg1 Rxe1  ++
+ 3  +2.44  00:00.005  1459         f7+ Kh8 fxe8=Q+ Qxe8 Kg1 Nxf2 Rxe8+ Bxe8 Kxf2
+ 4  +2.10  00:00.006  1918         f7+ Kxf7 Rxe4 Rb6 Re2 Rxe2 Bxe2
+ 5  +2.40  00:00.012  4554         f7+ Kxf7 Rxe4 Rb6 Bg2 Rxe4 Ng5+ Kg8 Nxe4
+ 6  +2.40  00:00.016  6369         f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Rc6
+ 7  +2.47  00:00.036  13990        f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Rc6 Kg1
+ 8- +1.72  00:00.051  19006        FL --
+ 8  +1.72  00:00.052  19420        f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Bc6 Ng5 Rb6
+ 9+ +2.47  00:00.058  21469        f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Bc6 Ng5 Rb6  ++
+ 9  +2.72  00:00.132  50019        f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Bc6 Ng5 Kh8 Nf7+ Kg8
+10  +2.48  00:00.169  62659        f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Bc6 Bg2 Rb6 Bg5 Bxe4 Bxe4
+11  +2.67  00:00.272  98649        f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Bc6 Ng5 Rb6 b3 Qe8 Rd3
+12  +2.80  00:00.766  276128       f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Bc6 Qf4 Kh8 b3 Bxe4 Bb2+ Kh7 Qxe4
+13  +2.76  00:01.245  451989       f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Bc6 Bd3 Kh8 Bg5 Rb6 Nf6 Rxb2 Qxb2
+14  +2.65  00:02.940  1055458      f7+ Kxf7 Rxe4 Rf8 Nh4 Rb6 Nxf5 gxf5
+15  +2.99  00:28.737  10301994     f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Bc6 Bd3 Qe6 Ng5 Qf6 c5 h4 g4 Ra8 cxd6 Qxd6+ Kg1
+16  +3.03  00:35.934  12793964     f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Bc6 Ng5 Rb6 b3 Nh6 Qf6 Ng4+ hxg4
+17  +3.04  00:47.526  17013639     f7+ Kxf7 Rxe4 Rxe4 Ng5+ Kg8 Nxe4 Bc6 Ng5 Rb6 b3 Qb8 <TT> Bd3 Bd7 Bxf5 Bxf5 Qf3 Rc6
 
-Remove "pure Newton's square root method" stuff and get math.h loaded.
+Read ./DEEPER_PRUNING.md for thoughts.
 
+Neural kings and pawns evaluator trained on EGTB + deep endgame position searches.