More aggressive move reductions; Wilson bound overrides static signal when available

SPRT playtest (2000 games, sd/depth matched): -535 =875 +590, score=0.514,
llr=+1.04 (floated around +1.0 the whole run, never went negative) toward
H1=+2.94. Not a confirmed win, but searches meaningfully deeper (avg depth
14.35 vs 13.50, EBF 3.131 vs 3.305) at no measurable strength cost -- worth
banking and continuing to accumulate games against.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 181 insertions, 40 deletions

diff --git a/src/searchsup.c b/src/searchsup.c
index d43d387..397d57e 100644
--- a/src/searchsup.c
+++ b/src/searchsup.c
@@ -19,6 +19,7 @@ Revision History:
 **/
 
 #include "chess.h"
+#include <math.h>  // log(), log10() -- _FhBasedExtraPliesF
 
 extern SCORE g_iRootScore[2];
 extern ULONG g_uHardExtendLimit;
@@ -856,6 +857,110 @@ _FlatTableExtraPlies(IN ULONG uRemainingDepth, IN ULONG uMoveNum)
     return (ULONG)(iExtraTicks / ONE_PLY);
 }
 
+// 2026-10-23: replaces _FlatTableExtraPlies as GetLMRReduction's primary
+// magnitude source (that function is kept as the JOINT_TABLE_SENTINEL
+// fallback below, for buckets the joint table has no opinion on at all).
+//
+// Derived from a direct regression against Crafty's and Stockfish's own
+// (independently play-tested) reduction schedules: at fixed remaining
+// depth, both engines' actual reduction in plies is extremely well
+// predicted (R^2=0.96 against Stockfish's own formula, checked across a
+// depth 2-13 x move 2-32 grid) by a single term proportional to
+// -log10(predictedFH%/100) -- i.e. both engines are implicitly already
+// reducing in proportion to how unlikely a move is to fail high, using
+// depth and move-number as cheap proxies for that probability. We have
+// a materially richer, directly-measured estimate of the same quantity
+// (uPredictedFH, 5 axes vs. their 2), so this leans on it directly
+// rather than re-deriving the same signal through depth/move-number
+// proxies -- the same way the existing admission gate already does.
+//
+// The proportionality constant itself (k) is NOT fixed -- it was also
+// measured, in the same regression, to climb with remaining depth
+// (k =~ 0.32*ln(depth) + 0.48 against Stockfish's schedule): both
+// engines reduce more per unit of -log10(FH%) the deeper the search
+// is. This isn't re-deriving something the FH% estimate already
+// contains -- FH% itself already depends on depth, but evidently not
+// with the magnitude a calibrated reduction schedule needs, so depth
+// earns a second, explicit role here beyond whatever it contributed to
+// uPredictedFH upstream.
+//
+// uRemainingDepth is deliberately capped at 13 plies before computing
+// k -- not an arbitrary safety margin, but the exact point where the
+// joint table's own raw collection data runs out (see the DEEPER_
+// PRUNING.md "leftover-population question" discussion: tries per
+// cell fall from 56,021 at depth 13 to low hundreds by depth 16-19 and
+// to zero past depth 19). Freezing k's depth input here means k stops
+// growing exactly where the data backing it stops being trustworthy --
+// and since _ShipDepthBucket() independently freezes uPredictedFH's
+// own depth axis at the same raw-13 boundary, both halves of this
+// formula plateau together, automatically capping total reduction to
+// a flat ceiling past this depth. That's the direct fix for the
+// "deterministic re-pruning across iterations" failure mode discussed
+// the same day: residual depth (nominal depth minus this flat cap)
+// diverges without bound as nominal search depth grows past 13, so a
+// move reduced this heavily is guaranteed to eventually receive a
+// residual look deep enough to find whatever it was missing, no
+// matter how deep the overall iterative-deepening search goes -- it
+// is never reduced by a *growing* amount indefinitely.
+//
+// Normalized against LMR_FH_BASELINE_PCT_THOUSANDTHS (the same 0.400%
+// whole-population baseline the dynamic-danger check already uses) so
+// an exactly-average leftover move earns ~0 extra plies (just the
+// existing flat -ONE_PLY base), and only moves predicted safer than
+// that baseline earn positive extra, scaling up continuously with how
+// much safer -- no threshold cliff, matching how neither reference
+// engine's own formula has an admission cliff either.
+// Relocated here (was originally right before CanWeReduceOrPruneMove)
+// so _FhBasedExtraPliesF below can use it -- same single definition,
+// just earlier in the file now. See its original comment, still in
+// place further down, for the full "0.400%" provenance story.
+#define LMR_FH_BASELINE_PCT_THOUSANDTHS (400)  // 0.400%
+
+// Returns the EXTRA reduction (beyond the base -ONE_PLY) in raw depth
+// ticks, rounded to the nearest QUARTER_PLY -- the engine's actual
+// reduction/extension resolution (chess.h: ONE_PLY=64, QUARTER_PLY=16,
+// i.e. depth is fixed-point in quarter-ply units throughout the
+// search, not whole plies). The regression this formula is built from
+// was fit in continuous plies; truncating to whole plies here would
+// throw away most of its resolution for no reason, so this returns
+// ticks directly instead of letting the caller round first.
+static ULONG
+_FhBasedExtraPlyTicks(IN ULONG uRemainingDepth, IN ULONG uPredictedFHThousandths)
+{
+    double dDepth = (double)(MINU(uRemainingDepth / ONE_PLY, 13));
+    double dK;
+    double dFhThousandths;
+    double dNegLogFh;
+    double dNegLogBaseline;
+    double dExtraPlies;
+    ULONG uTicks;
+
+    if (dDepth < 2.0)
+    {
+        dDepth = 2.0;
+    }
+    dK = 0.32 * log(dDepth) + 0.48;
+
+    dFhThousandths = (double)uPredictedFHThousandths;
+    if (dFhThousandths < 1.0)
+    {
+        dFhThousandths = 1.0;  // avoid log(0); table has no real 0 cells anyway
+    }
+    dNegLogFh = -log10(dFhThousandths / 100000.0);
+    dNegLogBaseline =
+        -log10((double)LMR_FH_BASELINE_PCT_THOUSANDTHS / 100000.0);
+
+    dExtraPlies = dK * (dNegLogFh - dNegLogBaseline);
+    if (dExtraPlies <= 0.0)
+    {
+        return 0;
+    }
+    // Round to nearest quarter-ply tick (ONE_PLY=64, QUARTER_PLY=16).
+    uTicks = (ULONG)((dExtraPlies * (double)ONE_PLY / (double)QUARTER_PLY) + 0.5)
+             * QUARTER_PLY;
+    return uTicks;
+}
+
 // data.c:1209's "Overall eligible FH%: 0.400%" -- the tries-weighted
 // fail-high rate of the WHOLE fPruneReduceEligible population across
 // the 500B-node joint-table collection run. Used two ways: (1) as the
@@ -879,7 +984,7 @@ _FlatTableExtraPlies(IN ULONG uRemainingDepth, IN ULONG uMoveNum)
 // rate is bimodal -- ~71% at exactly 0, with the next real mass not
 // starting until ~1%). So: intended behavior preserved, shipped
 // behavior changed by an immeasurably small amount.
-#define LMR_FH_BASELINE_PCT_THOUSANDTHS (400)  // 0.400%
+// (Defined earlier in this file now -- see _FhBasedExtraPliesF's use.)
 
 // 2026-10-02, one-off diagnostic support for chess.h's u64FHJointHist:
 // 8 attempts bands (0,1,2,3-4,5-9,10-24,25-99,100+) and 8 rate bands in
@@ -1070,7 +1175,7 @@ Return value:
     ASSERT(mv.uMove);
     ASSERT(!InCheck(&ctx->sPosition, ctx->sPosition.uToMove));
 
-    ULONG uExtraPlies = 0;
+    ULONG uExtraTicks = 0;
 #ifdef DISABLE_LMR
     // See GNUmakefile's DISABLE_LMR comment. Bail before any signal is
     // consulted so the no-LMR arm is a clean control: no reduction, and
@@ -1079,41 +1184,68 @@ Return value:
 #endif
 
     INT iLocalExtend;
-    FLAG fBaseEligible;
-    FLAG fDynamicDanger = (uAttempts > 0) &&
-        (uFHPctThousandths > LMR_FH_BASELINE_PCT_THOUSANDTHS);
+    ULONG uExtraTierForCounters;
 
+    // 2026-10-23: admission is no longer gated on the static table's
+    // opinion (LMR_STATIC_ADMIT_PCT_THOUSANDTHS retired from this
+    // function -- CanWeReduceOrPruneMove's base eligibility, i.e. "is
+    // this move even in the leftover tier," is still the only gate).
+    // Every eligible leftover move gets at least the base -ONE_PLY;
+    // uPredictedFH drives MAGNITUDE continuously via
+    // _FhBasedExtraPlyTicks below instead of gating whether to reduce
+    // at all -- matching how neither Crafty's nor Stockfish's own
+    // formula has an admission cliff either.
+    //
+    // 2026-10-23: the dynamic per-move-identity signal (this move's
+    // own observed fail-high history) no longer gets a separate
+    // veto-to-zero branch (the old fDynamicDanger check, removed).
+    // Deliberate, not an oversight -- discussed at length the same
+    // day: neither Crafty nor Stockfish exempts a move from reduction
+    // because of its own track record (Crafty has no dynamic signal
+    // in its reduction formula at all; Stockfish's analogous history-
+    // based term only ever feeds one continuous formula, never a
+    // separate exemption branch). A move's own real fail-high history
+    // is folded into the SAME formula's input instead, as a Wilson-
+    // score upper confidence bound (GetMoveFailHighWilsonUpperBoundThousandths,
+    // dynamic.c) on that move's true fail-high rate -- a tighter
+    // (lower) bound than the static table's prediction is real,
+    // sample-size-earned evidence of safety and is allowed to win;
+    // a looser (higher) bound -- including the "real history of
+    // failing high" case -- never overrides the static table's own
+    // reading, by construction: taking the minimum of two FH%
+    // estimates can only ever pick the smaller (safer) one, never
+    // push the effective rate up. (Deliberately NOT using this to
+    // justify reducing a dangerous-looking move LESS, either -- see
+    // the same discussion: the data argues searching a flagged-
+    // dangerous move deeper doesn't straightforwardly buy a safer
+    // read, since the table's own depth axis already shows shallower
+    // remaining depth correlates with HIGHER fail-high rates, not
+    // lower, so "reduce less" has no causal footing to stand on here.)
+    //
+    // Also 2026-10-23: works in raw depth ticks (QUARTER_PLY-rounded),
+    // not whole plies -- the engine's actual reduction/extension
+    // resolution is quarter-ply fixed-point throughout (chess.h:
+    // ONE_PLY=64, QUARTER_PLY=16), and the regression this formula
+    // comes from was fit in continuous plies, so truncating to whole
+    // plies here would throw away most of that precision for no
+    // reason. The old flat-table fallback (JOINT_TABLE_SENTINEL case)
+    // still only has whole-ply resolution, hence the *ONE_PLY there.
     if (JOINT_TABLE_SENTINEL == uPredictedFH)
     {
-        // No static data for this exact bucket, fall back on dynamic.
-        fBaseEligible = (FALSE == fDynamicDanger);
+        // No static data for this exact bucket -- fall back to the
+        // old flat Crafty-shaped (depth, moveNum) table rather than
+        // feeding a sentinel into the log formula. (Not blended with
+        // the Wilson bound either -- there's no static reading here
+        // to take a minimum against.)
+        uExtraTicks = _FlatTableExtraPlies(uRemainingDepth, uLegalMoves) * ONE_PLY;
     }
     else
     {
-        // Static table has an opinion: admission is now a wide-
-        // open, separate threshold from the extra-tier magnitude
-        // grading below (LMR_STATIC_ADMIT_PCT_THOUSANDTHS, not
-        // JOINT_EXTRA_TIER1_PCT_THOUSANDTHS) -- only veto the base
-        // reduction when the static table itself thinks this
-        // bucket is actively bad. A dynamic-danger reading does
-        // NOT veto this either -- it only caps uExtraPlies to 0
-        // below.
-        fBaseEligible = (uPredictedFH < LMR_STATIC_ADMIT_PCT_THOUSANDTHS);
-    }
-    if (FALSE == fBaseEligible)
-    {
-        if (NULL != puExtraTier)
-        {
-            *puExtraTier = 0;
-        }
-        return(0);
+        ULONG uWilsonBound = GetMoveFailHighWilsonUpperBoundThousandths(mv);
+        ULONG uEffectiveFH = MINU(uPredictedFH, uWilsonBound);
+        uExtraTicks = _FhBasedExtraPlyTicks(uRemainingDepth, uEffectiveFH);
     }
 
-    // We are going to reduce it; let's see if we want to pile on extra
-    // reduction because we're pretty sure it's "safe".
-    uExtraPlies = ((FALSE == fDynamicDanger)) ?
-        _FlatTableExtraPlies(uRemainingDepth, uLegalMoves) : 0;
-
     // Clamp.  Combine with the baseline -ONE_PLY and floor the total
     // so it never leaves less than JOINT_EXTRA_MIN_REMAINING_DEPTH
     // remaining -- never rely on the move loop's own unsigned-
@@ -1125,25 +1257,28 @@ Return value:
     // -ONE_PLY -- no need to fold a variable iExtend into it.
     ASSERT(0 == iExtend);
     iLocalExtend = -ONE_PLY;
-    if (uExtraPlies > 0)
+    uExtraTierForCounters = 0;
+    if (uExtraTicks > 0)
     {
         // MAX0 branchlessly floors the numerator at 0 before the
-        // divide (it can go negative when uRemainingDepth is small),
-        // so uMaxExtra is safely non-negative going into MINU -- both
-        // macros need their inputs' high-order bit clear, true here
-        // since none of uRemainingDepth/ONE_PLY/JOINT_EXTRA_MAX_PLIES
-        // come anywhere near 2^31.
+        // divide (it can go negative when uRemainingDepth is small).
         INT iExtraNumerator =
             (INT)uRemainingDepth - 2 * (INT)ONE_PLY -
             (INT)JOINT_EXTRA_MIN_REMAINING_DEPTH;
-        ULONG uMaxExtra = (ULONG)MAX0(iExtraNumerator) / (ULONG)ONE_PLY;
-        uExtraPlies = MINU(uExtraPlies, uMaxExtra);
-        uExtraPlies = MINU(uExtraPlies, (ULONG)JOINT_EXTRA_MAX_PLIES);
-        iLocalExtend -= (INT)(uExtraPlies * ONE_PLY);
+        ULONG uMaxExtraTicks = (ULONG)MAX0(iExtraNumerator);
+        uExtraTicks = MINU(uExtraTicks, uMaxExtraTicks);
+        uExtraTicks = MINU(uExtraTicks, (ULONG)(JOINT_EXTRA_MAX_PLIES * ONE_PLY));
+        iLocalExtend -= (INT)uExtraTicks;
+        // The joint-extra-tier instrumentation (u64JointExtraTries[]
+        // etc, JOINT_EXTRA_TIER_COUNT=6 slots, see chess.h) predates
+        // fractional extra plies and indexes by whole extra plies --
+        // floor to that resolution for counter purposes only; the
+        // actual applied reduction above keeps full tick precision.
+        uExtraTierForCounters = uExtraTicks / ONE_PLY;
     }
     if (NULL != puExtraTier)
     {
-        *puExtraTier = uExtraPlies;
+        *puExtraTier = uExtraTierForCounters;
     }
     return(iLocalExtend);
 }
@@ -2034,6 +2169,12 @@ Return value:
     {
         INC(ctx->sCounters.tree.u64LeftoverTries);
 
+#ifdef HASHCAP_COLLECTION
+        INC(ctx->sCounters.tree.u64HashCapTries
+            [pi->fHashMoveIsCapture ? 1 : 0]
+            [(NODE_CUT == pi->eNodeType) ? 1 : 0]);
+#endif
+
 #ifdef FH_TABLE_COLLECTION
         if (TRUE == fMoveIsPruneReduceEligible)
         {