NNUE search integration: FH% tables, NODE_TYPE refinement, TREEDUMP rewrite

- data.c/chess.h: widen the joint FH% table's late axis 16->17, dedicating
  the new bucket 0 to a whole-node ("will something here fail high at
  all") sample rather than sharing it with the first-leftover-move curve.
  Add an NNUE-eval counterpart table (g_uJointFHThousandths_NNUE) and an
  NNUE counterpart of the beta-side table, selected at runtime based on
  whether NNUE eval is active.
- search.c/searchsup.c: add _RefineNodeTypeGuess, which uses the FH%
  tables (via g_NodeTypeGuessTable) to refine a node's NODE_TYPE guess
  instead of the old heuristic; thread uParentMoveFHEstimate through
  PLY_INFO so a node can see its parent's prediction for the move that
  led to it.
- generate.c: initialize sKillers[2]/[3].uBonus unconditionally (an
  uninitialized-stack read caught by valgrind under LTO; harmless today
  but a real bug).
- dumptree.c/chess.h/GNUmakefile: replace the long-dead DUMP_TREE XML
  tree dump (written for a GUI this project no longer has) with TREEDUMP,
  a flat, path-keyed move/result trace meant to diff against an
  identically-instrumented Stockfish build -- see docs/TREEDUMP_FORMAT.md.
- docs/DEEPER_PRUNING.md: notes on closing the search-efficiency gap vs
  Stockfish.
- crafty_egtb/stockfish_xboard/typhoon_nnue: wrapper scripts so
  match_play.py's Engine class can drive Crafty, Stockfish (via
  PolyGlot), and an NNUE-enabled typhoon the same way it drives HCE
  typhoon.
- tests/lop: rename lop.txt -> lop.ep_; update compare_lop.cpp.
- annotate_pgns: add aggregate_marginsafety.py, dedup_pgn.py,
  fit_lazy_probe.py; update annotate.py/run_nnue_pipeline.py for the NNUE
  training pipeline.
- .gitignore: ignore compile_commands.json (clangd build DB) and
  eval.nnue (a symlink into the untracked head_reference/ comparison
  baseline), neither of which belongs in the repo.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 97 insertions, 58 deletions

diff --git a/src/search.c b/src/search.c
index 6f5e90c..c054333 100644
--- a/src/search.c
+++ b/src/search.c
@@ -18,29 +18,37 @@ Abstract:
     them.
 
     A type 2 node is either (a) a successor of any type-3 node, or,
-    (b) it's any successor (other than the first) of a type-1 node.
-    With perfect move ordering, the first branch at a type-2 node will
-    "refute" the move made at the previous ply via the alpha/beta
-    algorithm.  This node requires good move ordering, because you
-    have to find a move good enough that your opponent would not play
-    the move he chose that led to this position.  If you try a poor
-    move first, it won't produce a cutoff, and you have to search
-    another move (or more) until you find the "good" move that would
-    make him not play his move.
+    (b) it's any successor (other than the first) of a type-1 node.  A
+    type 2 node is also called a CUT node.  With perfect move
+    ordering, the first branch at a type-2 node will "refute" the move
+    made at the previous ply via the alpha/beta algorithm.  A
+    refutation (fail high) below a PV node indicates this move should
+    be searched with a full window -- it may be better than the move
+    on the PV here.  A fail high below a minimial window type 3 node
+    confirms its ancestor's hypothesis that this position is "too
+    good" and the opponent won't let the PV reach it (by choosing
+    another path).
 
-    A type-3 node follows a type-2 node.  Here, you have to try every
-    move at your disposal.  Since your opponent (at the previous ply)
-    has played a "strong" move (supposedly the "best" move) you are
-    going to have to try every move you have in an effort to refute
-    this move.  None will do so (unless your opponent tried some poor
-    move first due to incorrect move ordering).  Here, move ordering
-    is not worth the trouble, since the ordering won't let you avoid
-    searching some moves.  Of course, with the transposition /
-    refutation table, ordering might help you get more "hits" if your
-    table is not large enough...
+    Type 2 (CUT) nodes require good move ordering, because you have to
+    find a move good enough that your opponent would not play the move
+    he chose that led to this position.  If you try a poor move first,
+    it won't produce a cutoff, and you have to search another move (or
+    more) until you find the "good" move that would make him not play
+    his move.
+
+    A type-3 node (aka ALL node) follows a type-2 node.  Here, you
+    have to try every move at your disposal.  Since your opponent (at
+    the previous ply) has played a "strong" move (supposedly the
+    "best" move) you are going to have to try every move you have in
+    an effort to refute this move.  None will do so (unless your
+    opponent tried some poor move first due to incorrect move
+    ordering).  Here, move ordering is not worth the trouble, since
+    the ordering won't let you avoid searching some moves.  Of course,
+    with the transposition / refutation table, ordering might help you
+    get more "hits" if your table is not large enough...
 
     As you can see, at type-2 nodes you have to do good move ordering
-    to choose that "1" move (or to choose that one out of a very few)
+    to choose that one move (or to choose that one out of a very few)
     that is good enough to cause a cutoff, while avoiding choosing
     those that are no good.  At a type-1 node, the same thing applies.
     If you don't pick the best move first (take the moves at the root
@@ -48,11 +56,6 @@ Abstract:
     beta incorrectly, and thereby increase the size of the total tree
     by a *substantial* amount.
 
-    By the way, some authors call type-1 nodes "PV" nodes, type-2
-    nodes "CUT" nodes, and type-3 nodes "ALL" nodes.  These make it
-    easier to read, but, unfortunately, I "cut my teeth" on the
-    Knuth/Moore paper and think in terms of type 1,2,3."
-
                                                  --Bob Hyatt, r.g.c.c
 
 Author:
@@ -76,19 +79,32 @@ extern FLAG g_fCanSplit[MAX_PLY_PER_SEARCH];
 #define PREPARE_TO_TRY_MOVES  (2)
 #define TRY_GENERATED_MOVES   (3)
 
-// 2026-10-13: "skip trying null-move at all" on the iEval <= iAlpha
-// side (alpha-side table, EV-neutral -- the real move loop still runs
-// and finds the correct score either way; we only forgo the cheap
-// null-move probe on the rare occasion it would have worked). Units
-// are thousandths of a percent (same convention as everywhere else
-// here). 300 is the validated default -- swept against both halves
-// of ecm.ep_ at sd6/sd10, spot-checked at sd16, and against 1000
-// cross_sectional_1.ep_ positions; consistently in the plateau past
-// the knee with no solve-count cost anywhere tested. Still overridable
-// for further sweeping; 0 disables entirely. See
-// NULLMOVE_SKIP_CALIBRATION.md for the original methodology.
+// 2026-10-22: recalibrated from 300 to 36000. The old 300 was tuned
+// against a proxy quantity -- "will the first leftover move fail
+// high" -- read out of the late==1 bucket. As of the 2026-10-21 dual-
+// table collection run, this lookup's late==0 bucket is a genuine
+// whole-node "will THIS node fail high" sample (see
+// dual_table_run_nnue/README.md), whose magnitude is wildly different
+// (mean ~46.7% vs the old proxy's sub-1% regime), so 300 was
+// essentially never firing. Re-swept empirically (true node cost of a
+// skipped null-move search, including recursive null-under-null
+// effects, isn't derivable in closed form) via nodecount sweeps across
+// independent cross-sectional samples at sd8 (500 positions), sd11
+// (500 positions, independent draw), and sd12 (210 positions, 8+ men,
+// NNUE eval) -- all three agree on a broad plateau from roughly 30000
+// to 44000 thousandths (-0.7% to -3.7% total nodes vs. disabled),
+// decaying back toward baseline above ~44000. 36000 was the single
+// value scoring strongest or near-strongest across the most of these
+// independent, low-variance samples, so it was picked as the new
+// default; see dual_table_run_nnue/ and /tmp/typhoon/nullskip_sweep/
+// for the sweep data. (Narrower/deeper spot-checks at sd14-sd18 were
+// also attempted but were dominated by a small number of outlier
+// positions with wildly heavy-tailed node counts -- not usable
+// signal at the sample sizes tried; large-n shallower sweeps were the
+// reliable evidence here.) Still overridable for further sweeping; 0
+// disables entirely.
 #ifndef NULLMOVE_SKIP_FH_CEILING_THOUSANDTHS
-#define NULLMOVE_SKIP_FH_CEILING_THOUSANDTHS (300)
+#define NULLMOVE_SKIP_FH_CEILING_THOUSANDTHS (36000)
 #endif
 
 // 2026-10-02: the joint-table-driven graduated-LMR decision itself
@@ -145,6 +161,10 @@ _PostReduceDepthBucketEfpAligned(ULONG uNextDepthRaw)
 }
 #endif
 
+// _RefineNodeTypeGuess lives in searchsup.c now, next to
+// FHThousandthsToBand and the FH-prediction tables it consults --
+// see its own comment there.
+
 // EXPERIMENT: is history+continuation evidence predictive of a
 // countermove match's own FH%? See chess.h's CM_EVIDENCE_BUCKETS
 // comment. Buckets by log-ish bands rather than linear, since evidence
@@ -463,7 +483,6 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx,
     {
         goto end;
     }
-    DTEnterNode(ctx, uDepth, FALSE, iAlpha, iBeta);
     pi->eMoveScoringScheme = MOVE_SCORING_NORMAL;
 
     // Note: pi->fInCheck set by CommonSearchInit.
@@ -614,6 +633,20 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx,
                              ctx->sPlyInfo[ctx->uPly - 3].iEval);
     }
 
+    // Now that iEval (and the improvement signals derived from it) are
+    // known, refine CommonSearchInit's ancestor-only NODE_TYPE guess --
+    // see _RefineNodeTypeGuess's own comment. Must happen before
+    // anything below reads pi->eNodeType (the nullmove block's
+    // NODE_PV check, StartParallelSearch's split-eligibility gate,
+    // etc). Also captures the node-local predicted-FH% lookup
+    // _RefineNodeTypeGuess had to do anyway (whichever of the beta/
+    // alpha tables applies to this iEval/iAlpha/iBeta relationship) --
+    // the static-nullmove and nullmove-skip blocks below need that
+    // exact same number and must reuse uNodeLocalFH instead of
+    // re-consulting the data.c tables a second time per node.
+    ULONG uNodeLocalFH = _RefineNodeTypeGuess(ctx, iEval, iAlpha, iBeta, uDepth,
+                        iEnemyImprovement, uRawImproveB);
+
     // If the Eval call above was a full eval (i.e. not lazy or super
     // lazy), there are usable attack bitboards.  We'll consult them
     // below to determine if moves likely lose material but grab this
@@ -722,11 +755,10 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx,
         // Static nullmove pruning / reverse futility pruning?
         if (iEval > iBeta)
         {
-            ULONG uStaticNMDepthB = DepthBucket(uDepth / ONE_PLY);
-            ULONG uRawEnemyImproveB = ImproveBucket(iEnemyImprovement);
-
             INC(ctx->sCounters.tree.u64QuickNullMoves);
 #ifdef FH_TABLE_COLLECTION
+            ULONG uStaticNMDepthB = DepthBucket(uDepth / ONE_PLY);
+            ULONG uRawEnemyImproveB = ImproveBucket(iEnemyImprovement);
             // Collection builds record a genuine whole-node try here
             // (ground truth committed at end:) and must NOT take the
             // shortcut below -- it would skip the real move loop
@@ -747,11 +779,13 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx,
                 [uBetaCollectDepthB][uBetaCollectGapB]
                 [uBetaCollectOwnImpB][uBetaCollectEnemyImpB]);
 #else
-            ULONG uBetaPredictedFHThousandths = _BetaTablePredictedFHThousandths(
-                uStaticNMDepthB,
-            GapBetaBucket((SCORE)(iEval - iBeta)),
-            uRawImproveB,
-            uRawEnemyImproveB);
+            // uNodeLocalFH was already computed above by
+            // _RefineNodeTypeGuess via this exact same
+            // _BetaTablePredictedFHThousandths lookup (same depth/gap/
+            // improve bucket inputs, gated by the same iEval > iBeta
+            // branch) -- reuse it instead of consulting the table
+            // again.
+            ULONG uBetaPredictedFHThousandths = uNodeLocalFH;
             if ((uBetaPredictedFHThousandths != 0xFFFF) &&
                 (uBetaPredictedFHThousandths > 96000))
             {
@@ -766,9 +800,9 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx,
         }
         else if (iEval <= iAlpha)
         {
+#ifdef FH_TABLE_COLLECTION
             ULONG uNullSkipDepthB = DepthBucket(uDepth / ONE_PLY);
             ULONG uRawGapAlphaB = GapCpBucket((SCORE)(iAlpha - iEval));
-#ifdef FH_TABLE_COLLECTION
             // Whole-node sample for the alpha table's reserved
             // late==0 bucket (see chess.h) -- the alpha-side analog
             // of the beta-side whole-node try just above, covering
@@ -792,20 +826,19 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx,
                 [uAlpha0CollectDepthB][0][uAlpha0CollectGapB][0][uRawImproveB]);
 #endif
 #if (NULLMOVE_SKIP_FH_CEILING_THOUSANDTHS > 0)
-            {
-            USHORT uAlphaPredictedFH = _JointTablePredictedFHThousandths(
-                uNullSkipDepthB,
-            0,                  // hang bucket: assume nothing hung
-            uRawGapAlphaB,
-            1,                  // late bucket: "early move" proxy
-            uRawImproveB);
+            // uNodeLocalFH was already computed above by
+            // _RefineNodeTypeGuess via this exact same
+            // _JointTablePredictedFHThousandths lookup (same depth/
+            // hang/gap/late/improve bucket inputs, gated by the same
+            // iEval <= iAlpha branch) -- reuse it instead of
+            // consulting the table again.
+            USHORT uAlphaPredictedFH = (USHORT)uNodeLocalFH;
             if ((uAlphaPredictedFH != 0xFFFF) &&
                 (uAlphaPredictedFH < NULLMOVE_SKIP_FH_CEILING_THOUSANDTHS))
             {
                 fSkipNullForLowFH = TRUE;
                 INC(ctx->sCounters.tree.u64NullMoveSkippedLowFH);
             }
-            }
 #endif
         }
         if ((FALSE == fSkipNullForLowFH) &&
@@ -953,6 +986,7 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx,
                 // indexed when it's false, short-circuit preserved).
                 fNodeSplitEligible =
                     (MOVE_SCORING_IID_RESCORED != pi->eMoveScoringScheme) &&
+                    (NODE_ALL == ctx->sPlyInfo[ctx->uPly].eNodeType) &&
                     (uDepth >= ONE_PLY) &&
                     (TRUE == g_fCanSplit[uDepth / ONE_PLY - 1]) &&
                     (MOVE_COUNT(ctx, ctx->uPly) > 4);
@@ -1431,7 +1465,10 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx,
             uNullCalibRealLoopNodeCost;
     }
 #endif
-    DTLeaveNode(ctx, FALSE, iBestScore, mvBest);
+    if (!WE_SHOULD_STOP_SEARCHING)
+    {
+        TDResult(ctx, uDepth, iInitialAlpha, iBeta, pi->iEval, iBestScore);
+    }
     return(iBestScore);
 }
 
@@ -2378,7 +2415,10 @@ QSearch(IN SEARCHER_THREAD_CONTEXT *ctx,
     {
         goto end;
     }
-    DTEnterNode(ctx, 0, TRUE, iAlpha, iBeta);
+    // Note: deliberately not TREEDUMP-instrumented -- qsearch nodes
+    // vastly outnumber full-width nodes even at a shallow "sd"/"go
+    // depth" and would dominate a diff against Stockfish's own
+    // (likewise un-instrumented) qsearch; see TREEDUMP_FORMAT.md.
 
     // Probe interior node recognizers.  Qsearch runs at the deepest,
     // most performance-sensitive part of the tree, so this used to
@@ -2733,7 +2773,6 @@ QSearch(IN SEARCHER_THREAD_CONTEXT *ctx,
     ctx->sSearchFlags.fCouldStandPat[pos->uToMove] = fOrigStandPat;
     ASSERT(PositionsAreEquivalent(pos, &pi->sPosition));
     ASSERT(IS_VALID_SCORE(iBestScore) || WE_SHOULD_STOP_SEARCHING);
-    DTLeaveNode(ctx, TRUE, iBestScore, pi->mvBest);
 
     // Note: iBestScore can be +INFINITY_SCORE or -INFINITY_SCORE here even in the
     // absence of a legitimate mate detected if we disallowed stand