Dynamic move-ordering/LMR session: honest FH tracking, decoupled admission/magnitude, bugfixes, tooling

Lands the 2026-09-28..10-03 move-ordering investigation (full history in
lmr_testing/*.md). Summary of what ships:

LMR/move-ordering redesign (searchsup.c, dynamic.c, data.c, chess.h):
- GetLMRReduction's admission decision (do we reduce at all) is now
  decoupled from magnitude (how much): LMR_STATIC_ADMIT_THOUSANDTHS=500
  gates admission via the 4D joint fail-high table (data.c) plus a
  countermove-match veto; magnitude is a flat Crafty/Ethereal-shaped
  log(depth)*log(moves) table (g_iLMRQuietReduction), which measured as
  good or better than the old per-tier joint-table payout while being
  structurally immune to the per-bucket miscalibration that sank the
  tiered approach. g_FailHighs honesty fix (real 0/0 init, not a fake
  prior) underlies the dynamic-danger signal.
- Retired NumLeftoverMovesToSelect's depth-indexed selection budget --
  evidence showed the leftover pool has real findable signal a bailout
  was discarding.
- ShouldLatePrune (overt late-move pruning) implemented but disabled at
  its call site -- measured net-negative at the tested threshold; kept
  for reference and as a base for a future hang-bucket-9-targeted retry.
- PRUNE_VERIFY, PRUNE_LIVE, DISABLE_LMR: opt-in diagnostic build flags
  (GNUmakefile), off in the standard profile.

Bug fixes found and fixed while exercising this work:
- JOINT_EXTRA_TIER_COUNT 4->6 (chess.h): fixed an out-of-bounds counter
  write reachable at remaining depth >= 14 (unreachable at sd10, routine
  in real search/self-play).
- Node-type instrumentation (search.c, root.c, searchsup.c): fixed four
  compounding defects that produced impossible percentages (e.g. 106.2%
  of PV nodes) -- wrong denominator, qsearch misclassified as main
  search, untyped early-exit nodes, and fail-low tests using the
  in-place-raised alpha instead of the initial one.
- FH-rate unit unification (search.c, searchsup.c, dynamic.c, chess.h):
  GetMoveFailHigh* returned per-mille while g_uJointFHThousandths is
  thousandths-of-a-percent, two different units sharing a "_THOUSANDTHS"
  suffix. Unified to one scale, renamed to "_PCT_THOUSANDTHS".
  RateBucket's band edges fixed to match.
- QSearch option B (search.c): waives the opponent-stand-pat requirement
  at NODE_PV only (sound there since the exact score propagates; not
  sound at null-window nodes). iExtensionAmount unit fix (was 0/2 into a
  ONE_PLY-scaled field). pi->iEval now published when Eval() runs.
- eval_tune/match_play.py local_ebf: was reporting 0.000 in every match
  since 2026-09-18 (regex only matched the trailing-stats parser, not
  the live move-scan loop; engines were also spawned with "nopost" so
  root.c never emitted the lines at all). True per-ply EBF is ~2.0, not
  the ~3.9 every prior match report implied.
- eval_tune/test_vs_head.sh: positional-arg shift ordering bug caused
  every no-extra-args run to silently rebuild ./typhoon with a
  non-standard profile and, separately, could feed a flag's value in as
  the candidate binary path.
- PERF_COUNTERS cleanup (eval.c, search.c): moved 3 leaked rdtsc sites to
  the existing EVAL_TIME guard, deleted two concluded one-off diagnostics
  (u64HeadStyle*, u64FHJointHist/RateBucket/AttemptsBucket) and their
  report plumbing, and re-gated a per-move hang-material estimate behind
  FH_TABLE_COLLECTION instead of running it unconditionally.

Infrastructure:
- match_play.py time-odds support (--odds/--st-head/--st-candidate),
  validated against the engine seat rather than color.
- Wilson-upper-bound re-bake of the joint table (data.c): cells store a
  confidence-aware bound instead of a raw point estimate, so a consumer
  can threshold on the value alone; monotone, so every consumer became
  strictly more conservative. Verified behavior-neutral for LMR.

Known-open items (not resolved here, see lmr_testing/GET_LMR_REDUCTION.md
section 10 and 2026-10-03_UNCOMMITTED_TRIAGE.md): the self-play gate on
this candidate is underpowered (+5 Elo, 95% CI roughly [-6, +17]); the
curated-suite solve-count deltas used earlier in the investigation turned
out to be noise-dominated cancellation when diffed per position, not the
clean signal they first appeared to be.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 155 insertions, 11 deletions

diff --git a/src/root.c b/src/root.c
index 86d8b76..bc6e3be 100644
--- a/src/root.c
+++ b/src/root.c
@@ -469,11 +469,80 @@ Return value:
     Trace("Null move cutoff rate: %5.3f percent.\n",
           ((n / d) * 100.0));
 #ifdef PERF_COUNTERS
-    Trace("Node-type surprises: PV failed high/low=%llu, ALL failed high=%llu, "
-          "CUT failed low=%llu.\n",
-          ctx->sCounters.tree.u64PVNodeSurprised,
-          ctx->sCounters.tree.u64AllNodeFailedHigh,
-          ctx->sCounters.tree.u64CutNodeFailedLow);
+    // 2026-10-03: percentages added, each against the count of nodes of
+    // THAT TYPE (u64NodeTypeCount, incremented in CommonSearchInit),
+    // not against total nodes -- dividing by total nodes understates
+    // every rate by whatever share of the tree is some other type.
+    //
+    // 2026-10-03: this line now measures what its name says. Three
+    // separate defects were fixed to get there, all of which had been
+    // inflating it:
+    //   1. Denominator was total nodes, not nodes of that type.
+    //   2. Qsearch was included (~84% of the tree) -- a qsearch
+    //      stand-pat that beats beta looks like "ALL failed high" while
+    //      being the most ordinary event there. Now excluded from both
+    //      numerator (QSearch's copy of the counters was deleted) and
+    //      denominator (u64NodeTypeCount is incremented in Search()
+    //      only).
+    //   3. Nodes short-circuited by CommonSearchInit reached the
+    //      counters without ever having been typed for that visit, and
+    //      were scored against a stale eNodeType -- which produced an
+    //      impossible "106.2% of PV nodes". Now guarded by
+    //      fNodeTypedThisVisit.
+    // And a fourth in the fail-low test itself: it compared against the
+    // IN-PLACE-RAISED iAlpha rather than iInitialAlpha, so every node
+    // that successfully raised alpha was counted as a fail-low (the PV
+    // row read 99.1%). See search.c's comment at the counters.
+    {
+        double dPV = (double)ctx->sCounters.tree.u64NodeTypeCount[NODE_PV] + 1.0;
+        double dCUT = (double)ctx->sCounters.tree.u64NodeTypeCount[NODE_CUT] + 1.0;
+        double dALL = (double)ctx->sCounters.tree.u64NodeTypeCount[NODE_ALL] + 1.0;
+
+        Trace("Node-type surprises: PV failed high/low=%llu (%.1f%% of %.0f PV "
+              "nodes), ALL failed high=%llu (%.1f%% of %.0f ALL nodes), "
+              "CUT failed low=%llu (%.1f%% of %.0f CUT nodes).\n"
+              "  (full-width nodes only; qsearch excluded from both "
+              "numerator and denominator)\n",
+              ctx->sCounters.tree.u64PVNodeSurprised,
+              100.0 * (double)ctx->sCounters.tree.u64PVNodeSurprised / dPV, dPV - 1.0,
+              ctx->sCounters.tree.u64AllNodeFailedHigh,
+              100.0 * (double)ctx->sCounters.tree.u64AllNodeFailedHigh / dALL, dALL - 1.0,
+              ctx->sCounters.tree.u64CutNodeFailedLow,
+              100.0 * (double)ctx->sCounters.tree.u64CutNodeFailedLow / dCUT, dCUT - 1.0);
+
+        // Qsearch half, plus the PVS identity cross-check it exists to
+        // make possible: every CUT node that fails low forces its ALL
+        // parent to fail high, and that parent cannot have had an
+        // earlier child fail low (it would have cut off first), so the
+        // two should run ~1:1 ACROSS BOTH POPULATIONS. Any residual is
+        // ALL nodes resolved with no child searched at all --
+        // recognizer/EGTB fail-highs and null-move cutoffs.
+        {
+            double dQPV  = (double)ctx->sCounters.tree.u64QNodeTypeCount[NODE_PV] + 1.0;
+            double dQCUT = (double)ctx->sCounters.tree.u64QNodeTypeCount[NODE_CUT] + 1.0;
+            double dQALL = (double)ctx->sCounters.tree.u64QNodeTypeCount[NODE_ALL] + 1.0;
+            UINT64 u64AllFH = ctx->sCounters.tree.u64AllNodeFailedHigh +
+                              ctx->sCounters.tree.u64QNodeSurprise[NODE_ALL];
+            UINT64 u64CutFL = ctx->sCounters.tree.u64CutNodeFailedLow +
+                              ctx->sCounters.tree.u64QNodeSurprise[NODE_CUT];
+
+            Trace("  qsearch half: PV=%llu (%.1f%% of %.0f), ALL fh=%llu "
+                  "(%.1f%% of %.0f), CUT fl=%llu (%.1f%% of %.0f)\n",
+                  ctx->sCounters.tree.u64QNodeSurprise[NODE_PV],
+                  100.0 * (double)ctx->sCounters.tree.u64QNodeSurprise[NODE_PV] / dQPV,
+                  dQPV - 1.0,
+                  ctx->sCounters.tree.u64QNodeSurprise[NODE_ALL],
+                  100.0 * (double)ctx->sCounters.tree.u64QNodeSurprise[NODE_ALL] / dQALL,
+                  dQALL - 1.0,
+                  ctx->sCounters.tree.u64QNodeSurprise[NODE_CUT],
+                  100.0 * (double)ctx->sCounters.tree.u64QNodeSurprise[NODE_CUT] / dQCUT,
+                  dQCUT - 1.0);
+            Trace("  PVS identity (should be ~1:1): CUT-fails-low=%llu vs "
+                  "ALL-fails-high=%llu  ratio=%.2f\n",
+                  u64CutFL, u64AllFH,
+                  (double)u64AllFH / ((double)u64CutFL + 1.0));
+        }
+    }
 #endif
     if (ctx->sCounters.egtb.uProbes > 0)
     {
@@ -1270,7 +1339,8 @@ Return value:
 {
     ULONG uNumRootFailLows = 0;
     ULONG uDepth;
-    ULONG uFailState = 0;
+    ULONG uFailStateLo = 0;
+    ULONG uFailStateHi = 0;
     ULONG uColor;
     ULONG u;
     SCORE iAlpha = -INFINITY_SCORE;
@@ -1405,13 +1475,34 @@ Return value:
                 ASSERT(iScore == ctx->iRootScore);
                 ASSERT(mv.uMove);
                 ASSERT(SanityCheckMove(&ctx->sPosition, mv));
-                iAlpha = iScore - INITIAL_HALF_WINDOW;
-                iBeta = iScore + INITIAL_HALF_WINDOW;
+                //
+                // Don't center a cp-sized aspiration window on a mate
+                // score.  A mate found at this depth may be a search
+                // artifact (a defense pruned away by LMR/null-move/
+                // futility) that evaporates one ply deeper, in which case
+                // the next iteration's score isn't a nearby mate value at
+                // all -- it's an ordinary eval thousands of points away.
+                // That swing blows straight past SKIP_GRADUAL_OPENING and
+                // forces an immediate jump to the infinite window (and a
+                // full re-search of every root move) anyway, so just go
+                // wide on purpose instead of narrow-then-blown-open.
+                //
+                if ((ULONG)abs(iScore) >= NMATE)
+                {
+                    iAlpha = -INFINITY_SCORE;
+                    iBeta = +INFINITY_SCORE;
+                }
+                else
+                {
+                    iAlpha = iScore - INITIAL_HALF_WINDOW;
+                    iBeta = iScore + INITIAL_HALF_WINDOW;
+                }
                 g_iRootScore[uColor] = iScore;
                 g_iRootScore[FLIP(uColor)] = -iScore;
                 g_MoveTimer.bvFlags &= ~TIMER_RESOLVING_ROOT_FH;
                 g_MoveTimer.bvFlags &= ~TIMER_RESOLVING_ROOT_FL;
-                uFailState = 0;
+                uFailStateLo = 0;
+                uFailStateHi = 0;
                 (void)CheckTestSuiteMove(mv, iScore, uDepth);
                 break;
             }
@@ -1429,7 +1520,7 @@ Return value:
                 {
                     g_MoveTimer.bvFlags |= TIMER_MANY_ROOT_FLS;
                 }
-                _IterateWidenWindow(uColor, iScore, &iAlpha, &uFailState, -1);
+                _IterateWidenWindow(uColor, iScore, &iAlpha, &uFailStateLo, -1);
 
                 //
                 // Consider all moves again with wider window...
@@ -1451,7 +1542,7 @@ Return value:
                 ASSERT(mv.uMove);
                 ASSERT(SanityCheckMove(&ctx->sPosition, mv));
                 g_MoveTimer.bvFlags |= TIMER_RESOLVING_ROOT_FH;
-                _IterateWidenWindow(uColor, iScore, &iBeta, &uFailState, +1);
+                _IterateWidenWindow(uColor, iScore, &iBeta, &uFailStateHi, +1);
             }
         }
         while(1); // repeat until we run out of time or fall inside a..b
@@ -1507,6 +1598,59 @@ Return value:
         g_Options.u64CMEvidenceTries[u] = ctx->sCounters.tree.u64CMEvidenceTries[u];
         g_Options.u64CMEvidenceFH[u] = ctx->sCounters.tree.u64CMEvidenceFH[u];
     }
+    for (u = 0; u < HANG_CP_BUCKETS; u++)
+    {
+        g_Options.u64HangEstimateTries[u] = ctx->sCounters.tree.u64HangEstimateTries[u];
+        g_Options.u64HangEstimateFH[u] = ctx->sCounters.tree.u64HangEstimateFH[u];
+    }
+    g_Options.u64EligibleLeftoverTries = ctx->sCounters.tree.u64EligibleLeftoverTries;
+    g_Options.u64EligibleLeftoverFH = ctx->sCounters.tree.u64EligibleLeftoverFH;
+    for (u = 0; u < GAP_CP_BUCKETS; u++)
+    {
+        g_Options.u64GapEstimateTries[u] = ctx->sCounters.tree.u64GapEstimateTries[u];
+        g_Options.u64GapEstimateFH[u] = ctx->sCounters.tree.u64GapEstimateFH[u];
+    }
+    for (u = 0; u < MOVE_NUM_BUCKETS; u++)
+    {
+        g_Options.u64MoveNumEstimateTries[u] = ctx->sCounters.tree.u64MoveNumEstimateTries[u];
+        g_Options.u64MoveNumEstimateFH[u] = ctx->sCounters.tree.u64MoveNumEstimateFH[u];
+    }
+    for (u = 0; u < SEARCH_DEPTH_BUCKETS; u++)
+    {
+        g_Options.u64DepthEstimateTries[u] = ctx->sCounters.tree.u64DepthEstimateTries[u];
+        g_Options.u64DepthEstimateFH[u] = ctx->sCounters.tree.u64DepthEstimateFH[u];
+    }
+    memcpy(g_Options.u64JointTries, ctx->sCounters.tree.u64JointTries,
+           sizeof(g_Options.u64JointTries));
+    memcpy(g_Options.u64JointFH, ctx->sCounters.tree.u64JointFH,
+           sizeof(g_Options.u64JointFH));
+    memcpy(g_Options.u64JointExtraTries, ctx->sCounters.tree.u64JointExtraTries,
+           sizeof(g_Options.u64JointExtraTries));
+    memcpy(g_Options.u64JointExtraResearch, ctx->sCounters.tree.u64JointExtraResearch,
+           sizeof(g_Options.u64JointExtraResearch));
+    g_Options.u64LMPPruned = ctx->sCounters.tree.u64LMPPruned;
+#ifdef PRUNE_VERIFY
+    memcpy(g_Options.u64PruneVerifyTries, ctx->sCounters.tree.u64PruneVerifyTries,
+           sizeof(g_Options.u64PruneVerifyTries));
+    memcpy(g_Options.u64PruneVerifyFailHigh, ctx->sCounters.tree.u64PruneVerifyFailHigh,
+           sizeof(g_Options.u64PruneVerifyFailHigh));
+    memcpy(g_Options.u64PruneVerifyMarginSum, ctx->sCounters.tree.u64PruneVerifyMarginSum,
+           sizeof(g_Options.u64PruneVerifyMarginSum));
+    memcpy(g_Options.u64PruneVerifyPredSum, ctx->sCounters.tree.u64PruneVerifyPredSum,
+           sizeof(g_Options.u64PruneVerifyPredSum));
+    memcpy(g_Options.u64PruneVerifyDyn, ctx->sCounters.tree.u64PruneVerifyDyn,
+           sizeof(g_Options.u64PruneVerifyDyn));
+    memcpy(g_Options.u64PruneVerifyDynMiss, ctx->sCounters.tree.u64PruneVerifyDynMiss,
+           sizeof(g_Options.u64PruneVerifyDynMiss));
+#endif
+#ifdef JOINT_EXTRA_VERIFY
+    memcpy(g_Options.u64JointExtraVerifyTries, ctx->sCounters.tree.u64JointExtraVerifyTries,
+           sizeof(g_Options.u64JointExtraVerifyTries));
+    memcpy(g_Options.u64JointExtraVerifySilentMiss, ctx->sCounters.tree.u64JointExtraVerifySilentMiss,
+           sizeof(g_Options.u64JointExtraVerifySilentMiss));
+    memcpy(g_Options.u64JointExtraVerifyNullBailoutSignature, ctx->sCounters.tree.u64JointExtraVerifyNullBailoutSignature,
+           sizeof(g_Options.u64JointExtraVerifyNullBailoutSignature));
+#endif
 #ifdef TEST_NULL
     for (u = 0; u < NC_TOTAL_BUCKETS; u++)
     {