diff options
| author | Scott Gasch <[email protected]> | 2026-09-25 08:44:25 -0700 |
|---|---|---|
| committer | Scott Gasch <[email protected]> | 2026-09-25 08:44:25 -0700 |
| commit | 9b28080f8b3dca4588cb2aaaf748451f45e3c626 (patch) | |
| tree | 54e0aff5d6ee1520c590c7c04a5ae2f8299e7497 /src | |
| parent | 66740dc89c7351c120c25788d0079055034f4942 (diff) | |
Two ideas:
1. Nullmove R=3 + depth/6 along with some ShouldWeDoNullmoveSearch
tweaking and calibrating.
2. Probe EGTB everywhere in the search as long as the table/position
is resident in memory.
Diffstat (limited to 'src')
| -rw-r--r-- | src/CLAUDE.md | 76 | ||||
| -rwxr-xr-x | src/chess.h | 102 | ||||
| -rwxr-xr-x | src/eval_tune/test_vs_head.sh | 4 | ||||
| -rw-r--r-- | src/fathom/tbprobe.c | 236 | ||||
| -rw-r--r-- | src/fathom/tbprobe.h | 58 | ||||
| -rw-r--r-- | src/probe.c | 194 | ||||
| -rw-r--r-- | src/recogn.c | 234 | ||||
| -rw-r--r-- | src/root.c | 55 | ||||
| -rw-r--r-- | src/script.c | 137 | ||||
| -rw-r--r-- | src/search.c | 309 | ||||
| -rw-r--r-- | src/searchsup.c | 128 | ||||
| -rw-r--r-- | src/testrecogn.c | 273 |
12 files changed, 1629 insertions, 177 deletions
diff --git a/src/CLAUDE.md b/src/CLAUDE.md index b545edd..0bfc028 100644 --- a/src/CLAUDE.md +++ b/src/CLAUDE.md @@ -162,13 +162,75 @@ an xboard-protocol REPL on stdin/stdout. Key commands used during development: configure Syzygy tablebase lookup. **Default EGTB path is hardcoded** in `main.c:297` to `/zscratch/egtb`, used -whenever `--egtbpath` isn't passed. That directory currently has full 5-man -WDL+DTZ Syzygy coverage but is **missing the WDL half of its 6-man set** (only -`.rtbz`/DTZ files exist for 6-man combos) -- `init_tb()` in -`fathom/tbprobe.c:752` requires the WDL file to exist before registering a -combination at all, so every 6-man entry is silently skipped and the engine -reports "up to 5 men" even though 6-man `.rtbz` files are sitting right there. -Not a config issue; the fix is downloading the missing 6-man `.rtbw` files. +whenever `--egtbpath` isn't passed. That directory now has **full 5-man and +6-man WDL+DTZ Syzygy coverage** (the earlier gap -- 6-man `.rtbz`/DTZ files +present but the `.rtbw`/WDL half missing, so `init_tb()`'s +`fathom/tbprobe.c:752` WDL-file-required check silently skipped every 6-man +combination and the engine reported "up to 5 men" -- has been closed by +downloading the missing 6-man `.rtbw` files). `TB_LARGEST` should now read 6 +at startup. + +### Predicting whether a deep probe will be cheap (memory) vs. expensive (disk) + +Historically the engine only probed the EGTB shallow/near-root (`search.c`'s +`RecognLookup` call gated its 3rd arg, `fProbeEGTB`, on +`ctx->uPly <= (g_uIterateDepth / 2)`, and QSearch's call hardcoded `FALSE` +-- never probed at all in qsearch) specifically to avoid stalling the tree +search on a disk read near the leaves, where nodes are cheap and plentiful +and a single blocking read would be ruinously expensive relatively speaking. + +That blanket avoidance left value on the table: once a few pieces have +traded off near a leaf, a position is often *already* covered by a +tablebase whose file was mapped into memory by an earlier, shallower probe +this same search -- Fathom (`fathom/tbprobe.c`) lazily `mmap()`s + parses +each material signature's `.rtbw`/`.rtbz` file exactly once per process, +on first probe, behind a double-checked-locking `atomic_bool +be->ready[type]` (`probe_table`, `tbprobe.c:1778`). Once `ready[WDL]` is +true for a given material signature, subsequent probes of that signature +are just memory/page-cache lookups, not disk reads -- but the old +depth-based gate couldn't tell the difference between "warm" and "cold" +material and refused all deep probes uniformly. + +Fixed by adding `tb_wdl_is_resident()` (`fathom/tbprobe.c`, declared in +`fathom/tbprobe.h`) -- a non-blocking check that recomputes the same +material-signature key Fathom's own `calc_key()` uses (pure per-side +piece-type popcounts, no square/turn/ep info needed, so no board +conversion required) and reads `be->ready[WDL]` for that signature. No +lock, no file I/O, just a handful of popcounts + one atomic load; safe to +call from every leaf-ish node. Wrapped as `EgtbProbeLikelyResident(ctx)` +in `probe.c` (mirrors `ProbeEGTB`'s castling/fifty-move/cardinality +gating, so it never claims a probe is "likely resident" that `ProbeEGTB` +itself would refuse outright). `search.c`'s two `RecognLookup` call sites +now pass `(ctx->uPly <= (g_uIterateDepth / 2)) || EgtbProbeLikelyResident(ctx)` +(main search) and `EgtbProbeLikelyResident(ctx)` (QSearch, which +previously never probed at all) instead of the old ply-only / hardcoded +`FALSE` gates. + +**Caveat**: `ready[WDL] == true` only proves the file's header/index +metadata was parsed, not that every compressed data page this specific +position's lookup will touch is already resident in the OS page cache. +For the small 3-5 man files that distinction is essentially moot in +practice (the whole file tends to end up page-cached after the first +touch); for large 6-man files it's a weaker guarantee -- a `ready` +6-man table can still have cold pages. Not yet measured empirically +whether this matters in practice; if deep-probe latency spikes show up +in profiling, this is the first place to look. + +**Not yet done, a natural follow-up**: root-time warm-up -- proactively +touching (probing) WDL tables for material signatures reachable within a +few plies of likely trades from the root position, before the main +search starts, so `ready[WDL]` is already true for most of what the deep +search will actually encounter rather than depending on an earlier +shallow-ply probe in the *same* search to have warmed it first. + +**Also worth knowing**: `ProbeEGTB` wraps the actual `tb_probe_wdl` call +in a single process-wide spinlock (`g_uEgtbLock`, `probe.c`). Increasing +probe frequency deep in the tree (now possible via the above) increases +contention on that spinlock under `MP=1` multithreaded search even when +every probe individually is memory-cheap -- not yet measured whether this +becomes a bottleneck in practice; if throughput regresses under multiple +threads after this change, check spinlock contention before assuming a +disk-latency cause. ## Testing: the ECM tactical suite diff --git a/src/chess.h b/src/chess.h index 72289ad..ff218a9 100755 --- a/src/chess.h +++ b/src/chess.h @@ -941,6 +941,36 @@ MOVE_STACK; // size the evidence-bucket counter arrays. #define CM_EVIDENCE_BUCKETS (7) +// TEST_NULL calibration harness (search.c's null-move precision/recall/ +// EV measurement -- see _NullCalibBucketIndex's comment in search.c for +// the full design). Revised 2026-09-24: dropped fThreat/fSkipNull/ +// en-prise-or-trapped as bucket dimensions -- all three are already +// validated, non-tunable binary production gates (fThreat/fSkipNull +// gate the call to WeShouldTryNullmovePruning in search.c; en-prise/ +// trapped was shown NOT to predict false-positive risk, only recall), +// so the calibration probe now just requires fThreat==FALSE && +// fSkipNull==FALSE as part of structural eligibility instead of +// spending bucket space re-measuring them. Added a second depth +// dimension (outer, pre-reduction uDepth, alongside the null probe's +// own post-reduction uNullDepth) and a second material dimension +// (mover-only piece count -- TotalNonPawnCount(pos, pos->uToMove) -- +// alongside the existing MAX(white,black) count) -- both deliberately +// independent of the current SelectNullmoveRFactor R schedule, so a +// cost model fit from this data stays valid if R changes. +// +// Per-ply, 0..15 directly plus one "16+" overflow bucket -- fine enough +// to fit a cost-vs-depth growth curve (EBF-style) and extrapolate to +// depths never directly observed at the sd used for a given run. Used +// for both depth dimensions (same banding, reused). +#define NC_DEPTH_BUCKETS (17) +#define NC_DEFICIT_BUCKETS (6) // (iEval-iBeta) cp bands +// Non-pawn piece count bands: 0-1,2,3-5,6+. Used for both material +// dimensions (same banding, reused). +#define NC_MATERIAL_BUCKETS (4) +#define NC_TOTAL_BUCKETS \ + (NC_DEPTH_BUCKETS * NC_DEPTH_BUCKETS * NC_DEFICIT_BUCKETS * \ + NC_MATERIAL_BUCKETS * NC_MATERIAL_BUCKETS) + // ---------------------------------------------------------------------- // // Accumulators @@ -1018,11 +1048,35 @@ typedef struct _COUNTERS UINT64 u64NullMoves; UINT64 u64NullMoveSuccess; #ifdef TEST_NULL - UINT64 u64QuickNullSuccess; - UINT64 u64QuickNullDeferredSuccess; - UINT64 u64QuickNullFailures; - UINT64 u64AvoidNullSuccess; - UINT64 u64AvoidNullFailures; + // Precision/recall calibration for the null-move try/avoid + // decision. Indexed by _NullCalibBucketIndex() (search.c). + // Per cell: how often we attempted a (diagnostic, always-run) + // null-move probe, how often the probe itself failed high, how + // often the real move loop at that node *actually* failed high + // (ground truth, read off Search()'s own iBestScore vs iBeta at + // its "end:" label), and how often both agreed. From these four + // counts: precision = uBothFH/uNullFH, recall = uBothFH/uRealFH. + UINT64 u64NCAttempts[NC_TOTAL_BUCKETS]; + UINT64 u64NCNullFH[NC_TOTAL_BUCKETS]; + UINT64 u64NCRealFH[NC_TOTAL_BUCKETS]; + UINT64 u64NCBothFH[NC_TOTAL_BUCKETS]; + // Cost side of the try/skip decision: sum (not average -- divide + // by u64NCAttempts at report time) of the node cost of the + // diagnostic null probe itself, and of the immediately-following + // real first move's full-depth search (the "R=0" alternative), + // per bucket. Lets a report compute avg-nodes-under-null vs. + // avg-nodes-under-first-move and the try-vs-skip break-even + // this implies, given the bucket's own nullFH rate. + UINT64 u64NCNullProbeNodeSum[NC_TOTAL_BUCKETS]; + UINT64 u64NCFirstMoveNodeSum[NC_TOTAL_BUCKETS]; + // True total cost of however the real search actually resolved + // this node (one move, several moves, or production's own + // separate null-move success) -- node count from right after + // the diagnostic probe returns through this node's "end:". + // Strictly more honest than u64NCFirstMoveNodeSum for the + // try-vs-skip EV calc, which only approximates "cost of the + // real alternative" as move 1 alone. + UINT64 u64NCRealLoopNodeSum[NC_TOTAL_BUCKETS]; #endif UINT64 u64LazyEvals; UINT64 u64SuperLazyEvals; @@ -1652,6 +1706,15 @@ typedef struct _GAME_OPTIONS UINT64 u64LeftoverFH; UINT64 u64CMEvidenceTries[CM_EVIDENCE_BUCKETS]; UINT64 u64CMEvidenceFH[CM_EVIDENCE_BUCKETS]; +#ifdef TEST_NULL + UINT64 u64NCAttempts[NC_TOTAL_BUCKETS]; + UINT64 u64NCNullFH[NC_TOTAL_BUCKETS]; + UINT64 u64NCRealFH[NC_TOTAL_BUCKETS]; + UINT64 u64NCBothFH[NC_TOTAL_BUCKETS]; + UINT64 u64NCNullProbeNodeSum[NC_TOTAL_BUCKETS]; + UINT64 u64NCFirstMoveNodeSum[NC_TOTAL_BUCKETS]; + UINT64 u64NCRealLoopNodeSum[NC_TOTAL_BUCKETS]; +#endif CHAR szLogfile[SMALL_STRING_LEN_CHAR]; CHAR szEGTBPath[SMALL_STRING_LEN_CHAR]; CHAR szBookName[SMALL_STRING_LEN_CHAR]; @@ -4225,6 +4288,9 @@ void TestRecognExhaustiveKNKP(void); void +TestRecognExhaustiveKQKP(void); + +void TestRecognExhaustiveKPK(void); void @@ -4254,6 +4320,9 @@ ProbeEGTB(SEARCHER_THREAD_CONTEXT *ctx, SCORE *score); FLAG ProbeEGTBRoot(SEARCHER_THREAD_CONTEXT *ctx, MOVE *pmv, SCORE *piScore); +FLAG +EgtbProbeLikelyResident(SEARCHER_THREAD_CONTEXT *ctx); + void InitializeEGTB(void); @@ -4324,13 +4393,30 @@ GenerateRandomLegalSymetricPosition(POSITION *pos); #define RECOGN_LOWER (3) #define RECOGN_EGTB (4) +// +// EGTB probing mode for RecognLookup's 3rd argument -- trinary rather +// than a plain FLAG so "probe only if we believe the page is resident" +// is a distinct option from "always probe" and "never probe". Letting +// RecognLookup itself decide (rather than making the caller compute +// EgtbProbeLikelyResident up front, unconditionally, as a function +// argument) means the residency check -- which can now involve real +// mincore() syscalls for 6-man tables, see EgtbProbeLikelyResident's +// own comment -- is only ever paid on the branch that's actually about +// to decide whether to call ProbeEGTB, i.e. skipped entirely at every +// node one of the cheap hand-written interior recognizers already +// resolves. See CLAUDE.md's EGTB residency section. +// +#define EGTB_PROBE_NEVER (0) +#define EGTB_PROBE_ALWAYS (1) +#define EGTB_PROBE_IF_RESIDENT (2) + void InitializeInteriorNodeRecognizers(void); ULONG RecognLookup(SEARCHER_THREAD_CONTEXT *ctx, SCORE *piScore, - FLAG fProbeEGTB); + ULONG uEgtbProbeMode); // // Shared, ctx-free KP+KP* "unstoppable passer" static analysis -- @@ -4368,4 +4454,8 @@ ULONG _RecognizeKNKP(SEARCHER_THREAD_CONTEXT *ctx, SCORE *piScore); +ULONG +_RecognizeKQKP(SEARCHER_THREAD_CONTEXT *ctx, + SCORE *piScore); + #endif // CHESS diff --git a/src/eval_tune/test_vs_head.sh b/src/eval_tune/test_vs_head.sh index e937389..d6d9fba 100755 --- a/src/eval_tune/test_vs_head.sh +++ b/src/eval_tune/test_vs_head.sh @@ -48,9 +48,9 @@ python3 ./match_play.py \ ${HEAD} \ ${CANDIDATE} \ --pgn ../../pgn/twic_filtered.pgn \ - --games 20000 \ + --games 10000 \ --workers 12 \ - --st 1 \ + --st 4 \ --sprt --elo0 0 --elo1 5 \ --scratch /usr/local/tmp/typhoon_match_${LABEL} \ --log ${LABEL}.log \ diff --git a/src/fathom/tbprobe.c b/src/fathom/tbprobe.c index 5eac014..9cce02d 100644 --- a/src/fathom/tbprobe.c +++ b/src/fathom/tbprobe.c @@ -1550,6 +1550,24 @@ static bool init_table(struct BaseEntry *be, const char *str, int type) be->data[type] = data; +#ifndef _WIN32 + if (type == WDL && be->num <= 5) { + // 3-5 man WDL files are small (empirically <=20MB each, <=0.4GB total + // across every 3-5 man signature -- see CLAUDE.md's EGTB residency + // section), so instead of inferring residency later from whether this + // header parse has run (a guess that's only reliable because these + // files are small enough to end up fully cached anyway), just wire + // the whole mapping into RAM now. Once ready[WDL] is true for one of + // these signatures, tb_wdl_is_resident()'s "yes" becomes a guarantee + // instead of a proxy -- no mincore() needed for this tier at all. + if (mlock(data, (size_t)be->mapping[type]) != 0) { + perror("mlock"); + // Not fatal -- this signature's residency answer just reverts to + // being an inference (like the 6-man case) instead of a guarantee. + } + } +#endif + bool split = type != DTZ && (data[4] & 0x01); if (type == DTM) be->dtmLossOnly = data[4] & 0x04; @@ -1900,6 +1918,224 @@ static int probe_table(const Pos *pos, int s, int *success, const int type) return v; } +// Cheap, non-blocking check for whether the WDL table covering a given +// material signature is already mapped into memory (i.e. init_table()'s +// one-time mmap()+header-parse has already run for this piece +// combination, via some earlier probe_table() call this process). +// +// This deliberately duplicates probe_table()'s key computation and +// tbHash[] walk instead of calling it, so that callers can ask "would a +// probe of this position be fast?" from deep in the search tree without +// ever taking tbMutex or touching a byte of the underlying file -- it's +// a handful of popcounts plus one atomic_bool load. +// +// Returns: +// 1 the material signature has a WDL table and it's already resident +// (mapped + header parsed) -- a probe here should be near-free. +// 0 the material signature has a WDL table but it hasn't been +// touched by any probe yet this run -- a probe here will pay the +// one-time mmap()+parse cost (and is still lazily-mapped, so may +// also fault in fresh pages from disk for the data itself). +// -1 no WDL table covers this material signature at all (either it's +// outside the largest cardinality Fathom found on disk, or it's +// KvK / otherwise not tabulated). +// +// NOTE: a return of 1 only guarantees the table's *header and index +// metadata* are parsed, not that every compressed data page this +// specific position's lookup will touch is already in the OS page +// cache. For the small 3-5 man files that's essentially the same thing +// in practice (the whole file ends up cached after the first touch); +// for large 6-man files it's a weaker guarantee. See CLAUDE.md. +int tb_wdl_is_resident(uint64_t white, uint64_t black, uint64_t queens, + uint64_t rooks, uint64_t bishops, uint64_t knights, + uint64_t pawns) +{ + uint64_t key = popcount(white & queens) * PRIME_WHITE_QUEEN + + popcount(white & rooks) * PRIME_WHITE_ROOK + + popcount(white & bishops) * PRIME_WHITE_BISHOP + + popcount(white & knights) * PRIME_WHITE_KNIGHT + + popcount(white & pawns) * PRIME_WHITE_PAWN + + popcount(black & queens) * PRIME_BLACK_QUEEN + + popcount(black & rooks) * PRIME_BLACK_ROOK + + popcount(black & bishops) * PRIME_BLACK_BISHOP + + popcount(black & knights) * PRIME_BLACK_KNIGHT + + popcount(black & pawns) * PRIME_BLACK_PAWN; + if (key == 0ULL) return -1; // KvK: not a tabulated material signature + + int hashIdx = (int)(key >> (64 - TB_HASHBITS)); + while (tbHash[hashIdx].key && tbHash[hashIdx].key != key) + hashIdx = (hashIdx + 1) & ((1 << TB_HASHBITS) - 1); + if (!tbHash[hashIdx].ptr || + atomic_load_explicit(&tbHash[hashIdx].error, memory_order_relaxed)) + return -1; + + struct BaseEntry *be = tbHash[hashIdx].ptr; + return atomic_load_explicit(&be->ready[WDL], memory_order_relaxed) ? 1 : 0; +} + +#ifndef _WIN32 +// Page-residency predictor for the 6-man tier, where init_table() can't +// just mlock() the whole file (some signatures run into the hundreds of +// MB to low GB -- see CLAUDE.md). Every dereference below is gated by a +// mincore() check first, so a call to this function never itself blocks +// on disk I/O: it either confirms every page a real probe_table() lookup +// would touch is already resident (and, having just read them, returns a +// verified "yes"), or it stops at the first non-resident page and +// returns "no" without reading past it. +// +// A small direct-mapped cache remembers pages already confirmed resident +// so repeated probes into the same hot region of a table don't re-pay a +// mincore() syscall every time. It deliberately only caches "yes" +// answers -- a page that mincore() reports as not-yet-resident is always +// re-checked for real next time, so a page that warms up between calls +// is picked up on its next probe rather than being stuck on a stale "no". +#define TB_RESIDENCY_PAGE_SIZE 4096u +#define TB_RESIDENCY_CACHE_BITS 12 +#define TB_RESIDENCY_CACHE_SIZE (1u << TB_RESIDENCY_CACHE_BITS) +static atomic_uintptr_t g_ResidencyCache[TB_RESIDENCY_CACHE_SIZE]; + +static size_t _ResidencyCacheSlot(uintptr_t pageAddr) +{ + uintptr_t h = (pageAddr >> 12) * 2654435761u; + return (size_t)(h & (TB_RESIDENCY_CACHE_SIZE - 1)); +} + +static int _PageIsResident(uintptr_t pageAddr) +{ + size_t slot = _ResidencyCacheSlot(pageAddr); + if (atomic_load_explicit(&g_ResidencyCache[slot], memory_order_relaxed) == pageAddr) + return 1; + + unsigned char vec = 0; + if (mincore((void *)pageAddr, TB_RESIDENCY_PAGE_SIZE, (char *)&vec) != 0) + return 0; // shouldn't happen for a page inside a live mapping; be conservative + + if (vec & 1) { + atomic_store_explicit(&g_ResidencyCache[slot], pageAddr, memory_order_relaxed); + return 1; + } + return 0; +} + +static int _RangeIsResident(const void *addr, size_t len) +{ + uintptr_t start = (uintptr_t)addr & ~(uintptr_t)(TB_RESIDENCY_PAGE_SIZE - 1); + uintptr_t end = ((uintptr_t)addr + len + TB_RESIDENCY_PAGE_SIZE - 1) & + ~(uintptr_t)(TB_RESIDENCY_PAGE_SIZE - 1); + for (uintptr_t p = start; p < end; p += TB_RESIDENCY_PAGE_SIZE) + if (!_PageIsResident(p)) + return 0; + return 1; +} + +// Precise, non-blocking "would a real WDL probe of this exact position +// hit resident pages only?" check. Unlike tb_wdl_is_resident() (which +// only knows whether a material signature's header has been parsed at +// all), this computes the same indexTable/sizeTable/data-block addresses +// decompress_pairs() would touch for this specific position and confirms +// each one is actually in the OS page cache before trusting the next. +// +// RETURN: 1 if a subsequent real probe is expected to be fully resident +// (no page fault / disk read), 0 otherwise (table not loaded, material +// untabulated, or some page on the lookup path isn't resident yet). +int tb_wdl_probe_is_resident(uint64_t white, uint64_t black, uint64_t kings, + uint64_t queens, uint64_t rooks, + uint64_t bishops, uint64_t knights, + uint64_t pawns, bool turn) +{ + Pos pos = { white, black, kings, queens, rooks, bishops, knights, pawns, + 0, 0, turn }; + + uint64_t key = calc_key(&pos, false); + if (key == 0ULL) + return 0; + + int hashIdx = (int)(key >> (64 - TB_HASHBITS)); + while (tbHash[hashIdx].key && tbHash[hashIdx].key != key) + hashIdx = (hashIdx + 1) & ((1 << TB_HASHBITS) - 1); + if (!tbHash[hashIdx].ptr || + atomic_load_explicit(&tbHash[hashIdx].error, memory_order_relaxed)) + return 0; + + struct BaseEntry *be = tbHash[hashIdx].ptr; + if (!atomic_load_explicit(&be->ready[WDL], memory_order_relaxed)) + return 0; + + // 3-5 man WDL files are mlock()'d whole at load time (init_table()) -- + // ready[WDL] already IS the guarantee for this tier, no page check + // needed. + if (be->num <= 5) + return 1; + + bool bside, flip; + if (!be->symmetric) { + flip = key != be->key; + bside = (pos.turn == WHITE) == flip; + } else { + flip = pos.turn != WHITE; + bside = false; + } + + struct EncInfo *ei = first_ei(be, WDL); + int p[TB_PIECES]; + size_t idx; + + if (!be->hasPawns) { + ei = &ei[bside]; + for (int i = 0; i < be->num;) + i = fill_squares(&pos, ei->pieces, flip, 0, p, i); + idx = encode_piece(p, ei, be); + } else { + int i = fill_squares(&pos, ei->pieces, flip, flip ? 0x38 : 0, p, 0); + int t = leading_pawn(p, be, FILE_ENC); + ei = &ei[t + 4 * bside]; + while (i < be->num) + i = fill_squares(&pos, ei->pieces, flip, flip ? 0x38 : 0, p, i); + idx = encode_pawn_f(p, ei, be); + } + + struct PairsData *d = ei->precomp; + if (!d->idxBits) + return 1; // constant-value table -- no data page to check at all + + uint32_t mainIdx = (uint32_t)(idx >> d->idxBits); + uint8_t *idxEntry = d->indexTable + 6 * (size_t)mainIdx; + if (!_RangeIsResident(idxEntry, 6)) + return 0; + + uint32_t block; + memcpy(&block, idxEntry, sizeof(block)); + block = from_le_u32(block); + uint16_t idxOffset = *(uint16_t *)(idxEntry + 4); + + int litIdx = (int)((idx & (((size_t)1 << d->idxBits) - 1)) - + ((size_t)1 << (d->idxBits - 1))); + litIdx += from_le_u16(idxOffset); + + if (litIdx < 0) { + while (litIdx < 0) { + uint32_t prev = block - 1; + if (!_RangeIsResident(&d->sizeTable[prev], sizeof(uint16_t))) + return 0; + block = prev; + litIdx += d->sizeTable[block] + 1; + } + } else { + if (!_RangeIsResident(&d->sizeTable[block], sizeof(uint16_t))) + return 0; + while (litIdx > d->sizeTable[block]) { + litIdx -= d->sizeTable[block] + 1; + block++; + if (!_RangeIsResident(&d->sizeTable[block], sizeof(uint16_t))) + return 0; + } + } + + uint8_t *dataBlock = d->data + ((size_t)block << d->blockSize); + return _RangeIsResident(dataBlock, sizeof(uint32_t) * 2); +} +#endif // !_WIN32 + static int probe_wdl_table(const Pos *pos, int *success) { return probe_table(pos, 0, success, WDL); diff --git a/src/fathom/tbprobe.h b/src/fathom/tbprobe.h index 82850e7..6f4e5e0 100644 --- a/src/fathom/tbprobe.h +++ b/src/fathom/tbprobe.h @@ -181,6 +181,64 @@ bool tb_init(const char *_path); void tb_free(void); /* + * Cheap, non-blocking check for whether the WDL table covering a given + * material signature (per-side piece-type bitboards, no square/turn/ep + * info needed) is already mapped into memory from an earlier probe this + * process -- i.e. whether a probe of a position with this material is + * likely to be low-latency versus pay a first-touch mmap()+parse (and + * possibly disk-read) cost. See the routine description in tbprobe.c for + * the full contract and caveats. + * + * PARAMETERS: + * - white, black, queens, rooks, bishops, knights, pawns: + * Per-side piece-type bitboards (kings/turn/ep are irrelevant to the + * material-signature key and are omitted). + * + * RETURN: + * - 1: table exists and is already resident (fast probe expected). + * - 0: table exists but hasn't been touched yet this run (first probe + * will pay the one-time mmap()+header-parse cost). + * - -1: no WDL table covers this material signature at all. + */ +int tb_wdl_is_resident(uint64_t white, uint64_t black, uint64_t queens, + uint64_t rooks, uint64_t bishops, uint64_t knights, + uint64_t pawns); + +/* + * Precise, non-blocking check for whether a WDL probe of this EXACT + * position (not just its material signature) would hit only pages + * already resident in the OS page cache. Unlike tb_wdl_is_resident() + * above -- which only knows whether the table's header/index metadata + * has been parsed at all, a weak proxy for large 6-man files -- this + * walks the same indexTable/sizeTable/compressed-data-block addresses a + * real probe would touch and confirms each one via mincore() before + * trusting the next, so it never blocks on disk I/O itself. Small (3-5 + * man) WDL files are mlock()'d whole at load time (see init_table() in + * tbprobe.c), so for that tier this degenerates to the same + * header-parsed check as tb_wdl_is_resident() -- a guarantee, not a + * guess, for that tier specifically. + * + * PARAMETERS: + * - white, black, kings, queens, rooks, bishops, knights, pawns: + * The current position (bitboards), matching tb_probe_wdl's layout. + * - turn: true if White is to move. + * + * RETURN: + * - 1: a real probe here is expected to touch only resident pages. + * - 0: table not loaded, material untabulated, or some page on the + * lookup path isn't resident yet. + * + * Not available under _WIN32 (mincore() has no portable equivalent + * wired up there yet). + */ +#ifndef _WIN32 +int tb_wdl_probe_is_resident(uint64_t white, uint64_t black, uint64_t kings, + uint64_t queens, uint64_t rooks, + uint64_t bishops, uint64_t knights, + uint64_t pawns, bool turn); +#endif + +/* * Probe the Win-Draw-Loss (WDL) table. * * PARAMETERS: diff --git a/src/probe.c b/src/probe.c index 2bddf99..269a1b9 100644 --- a/src/probe.c +++ b/src/probe.c @@ -138,10 +138,25 @@ Routine description: Build the set of bitboards Fathom's tb_probe_wdl wants out of our POSITION. Fathom expects the standard tablebase square numbering - (a1 == bit 0 ... h8 == bit 63), which is exactly what our TO64() - macro produces -- note this is *not* the same square numbering as - our engine's own internal BBSQUARE/COOR_TO_BB bitboards, so we - must not use those macros here. + (a1 == bit 0 ... h8 == bit 63); our own internal bitboards + (bbPieces/bbPawns/bbOccupiedSide) instead index bit = rankTopDown*8 + + file (rankTopDown 0 == rank 8 ... 7 == rank 1 -- see TO64's own + (7 - (x>>4)) flip). Those two numberings differ by exactly one + 64-bit byte-swap: internal bit r*8+f (byte r, offset f) lands at + Fathom bit (7-r)*8+f (byte 7-r, offset f) under bswap64, matching + TO64's formula term for term. So this can OR our own bitboards + together and byte-swap the result directly -- O(1) per output + bitboard -- instead of walking every piece one at a time through + CoorFromBitBoardRank8ToRank1/TO64. (2026-09-24: replaced the + original per-piece-popping version, which showed up as a real + self-play NPS/depth regression once EgtbProbeLikelyResident started + calling this from QSearch's hot path -- see CLAUDE.md.) + + No bbPieces[color][KING] exists (deliberately -- see its own + comment), so the king bitboard is built from the single square in + cKingSq[color] instead; equivalent to bswapping a bitboard with only + that one bit set, since bswap64 is just a bit permutation and + commutes with OR. Parameters: @@ -154,75 +169,22 @@ Return value: **/ { - ULONG uColor; - COOR c; - UINT64 bb; - - *pu64White = *pu64Black = 0ULL; - *pu64Kings = *pu64Queens = *pu64Rooks = 0ULL; - *pu64Bishops = *pu64Knights = *pu64Pawns = 0ULL; - - for (uColor = BLACK; uColor <= WHITE; uColor++) - { - UINT64 *pu64Side = (uColor == WHITE) ? pu64White : pu64Black; - - { - BITBOARD bbPawns = pos->bbPawns[uColor]; - - while (IS_ON_BOARD(c = CoorFromBitBoardRank8ToRank1(&bbPawns))) - { - bb = (1ULL << (TO64(c))); - *pu64Side |= bb; - *pu64Pawns |= bb; - } - } - - { - c = KingPosition(pos, uColor); - ASSERT(IS_ON_BOARD(c)); - ASSERT(IS_KING(pos->rgSquare[c].pPiece)); - bb = (1ULL << (TO64(c))); - *pu64Side |= bb; - *pu64Kings |= bb; - } - - { - BITBOARD bbType; - COOR cType; - - bbType = pos->bbPieces[uColor][KNIGHT]; - while (IS_ON_BOARD(cType = CoorFromBitBoardRank8ToRank1(&bbType))) - { - bb = (1ULL << (TO64(cType))); - *pu64Side |= bb; - *pu64Knights |= bb; - } + UINT64 u64Kings = (1ULL << COOR_TO_BIT_NUMBER(pos->cKingSq[WHITE])) | + (1ULL << COOR_TO_BIT_NUMBER(pos->cKingSq[BLACK])); - bbType = pos->bbPieces[uColor][BISHOP]; - while (IS_ON_BOARD(cType = CoorFromBitBoardRank8ToRank1(&bbType))) - { - bb = (1ULL << (TO64(cType))); - *pu64Side |= bb; - *pu64Bishops |= bb; - } - - bbType = pos->bbPieces[uColor][ROOK]; - while (IS_ON_BOARD(cType = CoorFromBitBoardRank8ToRank1(&bbType))) - { - bb = (1ULL << (TO64(cType))); - *pu64Side |= bb; - *pu64Rooks |= bb; - } - - bbType = pos->bbPieces[uColor][QUEEN]; - while (IS_ON_BOARD(cType = CoorFromBitBoardRank8ToRank1(&bbType))) - { - bb = (1ULL << (TO64(cType))); - *pu64Side |= bb; - *pu64Queens |= bb; - } - } - } + *pu64White = __builtin_bswap64((UINT64)pos->bbOccupiedSide[WHITE]); + *pu64Black = __builtin_bswap64((UINT64)pos->bbOccupiedSide[BLACK]); + *pu64Kings = __builtin_bswap64(u64Kings); + *pu64Queens = __builtin_bswap64( + (UINT64)(pos->bbPieces[WHITE][QUEEN] | pos->bbPieces[BLACK][QUEEN])); + *pu64Rooks = __builtin_bswap64( + (UINT64)(pos->bbPieces[WHITE][ROOK] | pos->bbPieces[BLACK][ROOK])); + *pu64Bishops = __builtin_bswap64( + (UINT64)(pos->bbPieces[WHITE][BISHOP] | pos->bbPieces[BLACK][BISHOP])); + *pu64Knights = __builtin_bswap64( + (UINT64)(pos->bbPieces[WHITE][KNIGHT] | pos->bbPieces[BLACK][KNIGHT])); + *pu64Pawns = __builtin_bswap64( + (UINT64)(pos->bbPawns[WHITE] | pos->bbPawns[BLACK])); } @@ -324,6 +286,72 @@ Return value: } +FLAG +EgtbProbeLikelyResident(SEARCHER_THREAD_CONTEXT *ctx) +/** + +Routine description: + + Cheap, non-blocking predictor for "would an EGTB probe of the + current position be low latency?" -- used to let deep search/qsearch + nodes probe the EGTB (which is normally reserved for shallow, near- + root plies to avoid stalling the tree search on a disk read) once + the specific pages this exact position's lookup needs are already + resident, whether from an earlier probe this run or (for 3-5 man + tables) because they were mlock()'d in whole at load time. + + Does not itself probe the tablebase, take g_uEgtbLock, or read any + file data that isn't already confirmed resident first -- see + tb_wdl_probe_is_resident in fathom/tbprobe.c for the mincore()-gated + walk this performs (indexTable/sizeTable/data-block addresses for + 6-man tables; a single ready[] check for the mlock()'d 3-5 man + tier). Safe to call from every leaf-ish node without worrying about + search slowdown. + + Same castling/fifty-move/cardinality gating as ProbeEGTB (a probe + that ProbeEGTB itself would refuse is never "likely resident"). + +Parameters: + + SEARCHER_THREAD_CONTEXT *ctx + +Return value: + + FLAG -- TRUE iff a subsequent ProbeEGTB call for this exact position + is expected to hit already-resident tablebase data rather than pay a + first-touch mmap()/parse (and possibly disk-read) cost. + +**/ +{ + POSITION *pos = &(ctx->sPosition); + UINT64 u64White, u64Black, u64Kings, u64Queens; + UINT64 u64Rooks, u64Bishops, u64Knights, u64Pawns; + ULONG wcount, bcount; + + if (FALSE == g_fEgtbInitialized || 0 == TB_LARGEST) + { + return(FALSE); + } + if ((0 != pos->bvCastleInfo) || (0 != pos->uFifty)) + { + return(FALSE); + } + wcount = TotalNonPawnCount(pos, WHITE) + FastCountBits(pos->bbPawns[WHITE]); + bcount = TotalNonPawnCount(pos, BLACK) + FastCountBits(pos->bbPawns[BLACK]); + if ((wcount + bcount) > TB_LARGEST) + { + return(FALSE); + } + + _BuildFathomBitboards(pos, &u64White, &u64Black, &u64Kings, &u64Queens, + &u64Rooks, &u64Bishops, &u64Knights, &u64Pawns); + return(1 == tb_wdl_probe_is_resident(u64White, u64Black, u64Kings, + u64Queens, u64Rooks, u64Bishops, + u64Knights, u64Pawns, + (pos->uToMove == WHITE))); +} + + // // Convert a Fathom tablebase square index (a1==0 ... h8==63) back into // our own COOR encoding. Inverse of TO64(). @@ -487,8 +515,28 @@ Return value: (mv.cTo == cTo) && (mv.pPromoted == pPromotes)) { + // Root-only: use the DTZ ply count tb_probe_root already + // packed into uResult (no extra probe) to report a real, + // ply-distance-aware mate score instead of the flat + // EGTB_WIN_SCORE/LOSS_SCORE used by in-search probes + // (ProbeEGTB/RecognLookup) -- those stay WDL-only and + // untouched. util.c's SCORE-to-string formatter already + // treats any |score| >= NMATE as a mate score and derives + // the move count as (INFINITY_SCORE - iScore) / 2, so + // INFINITY_SCORE -/+ uDtz round-trips through that + // unmodified as "+MATE<n>"/"-MATE<n>". Clamp defensively: + // a uDtz large enough to push the score below NMATE would + // just silently fall back to plain-decimal display, not + // corrupt anything, but keep it a real mate score at every + // legal DTZ value we could plausibly see (TB_MAX_PLY=256 + // exceeds NMATE's 200-point margin in principle, however + // unlikely in practice for <=7-man tables). + ULONG uDtz = TB_GET_DTZ(uResult); + if (uDtz >= (+INFINITY_SCORE - NMATE)) uDtz = (+INFINITY_SCORE - NMATE - 1); *pmv = mv; - *piScore = (TB_WIN == uWdl) ? EGTB_WIN_SCORE : EGTB_LOSS_SCORE; + *piScore = (TB_WIN == uWdl) ? + (SCORE)(+INFINITY_SCORE - uDtz) : + (SCORE)(-INFINITY_SCORE + uDtz); return(TRUE); } } diff --git a/src/recogn.c b/src/recogn.c index 2219e48..427297b 100644 --- a/src/recogn.c +++ b/src/recogn.c @@ -719,6 +719,169 @@ Return value: } +ULONG +_RecognizeKQKP(IN SEARCHER_THREAD_CONTEXT *ctx, + IN OUT SCORE *piScore) +/** + +Routine description: + + Recognize KQKP positions: exactly one queen vs exactly one pawn. + Deliberately narrow -- only handles the classically UNCONDITIONAL + win subset: the pawn has NOT yet reached the rank immediately + before queening (rank 7 for a White pawn, rank 2 for a Black one), + and the queen isn't hanging to the weak king or weak pawn (see + below). Once the weak pawn reaches that last rank it threatens to + promote into a second queen -- uncontested if nothing stops it, or + even a capturing promotion if it lands on the strong queen's square + -- either of which can rebalance material into a generic (and often + drawn) queen-vs-queen ending, regardless of which file the pawn is + on. That whole family of positions returns UNRECOGNIZED and falls + through to search/EGTB, same as every other case this function + doesn't recognize. + + 2026-09-24, in order of discovery (all via TestRecognExhaustiveKQKP, + each one found immediately after "fixing" the previous): + + First hole -- ~10% wrong across the enumeration, all one shape: the + queen placed adjacent to the weak king, undefended, with the weak + side to move (e.g. K1kQ4/p7/8/8/8/8/8/8 b - - 0 0 -- White Ka8, + Black Kc8, White Qd8, Black Pa7, Black to move: ...Kxd8 wins the + queen for free). The original version of this function never + considered queen safety at all. + + Second hole, found immediately after fixing the first (king- + adjacency only got the wrong rate down to ~4%, not to zero): the + WEAK PAWN capturing the queen while promoting, e.g. + k1K1q3/3P4/8/8/8/8/8/8 w - - 0 0 (Black Ka8/Qe8, White Kc8/Pd7, + White to move) -- dxe8=Q captures Black's queen and promotes in the + same move, which EGTB confirms is winning for White even though + White has the pawn. Fixed by also checking whether the queen sits + on a square the weak pawn attacks (g_PawnAttackOriginBB), not just + king adjacency, and by dropping the earlier "unless defended by the + strong king" carve-out entirely for both threats -- a pawn capture- + promotion is catastrophic even if recaptured (nets down to a bare- + kings draw at best, never the claimed win). + + Third hole, found immediately after fixing the second: the original + rook/bishop-file-only exclusion for a pawn on the queening-adjacent + rank missed the exact same "uncontested promotion creates a second + queen" problem on every OTHER file too, e.g. + k1K2q2/3P4/8/8/8/8/8/8 w - - 0 0 (White's d-pawn just pushes to + d8=Q, an ordinary center-file pawn, and EGTB says the resulting + KQKQ position is a draw, not the claimed win) and + K1k5/Q7/8/8/8/8/1p6/8 b - - 0 0 (Black's b2 knight-file pawn simply + promotes THIS move, ...b1=Q, queen too far away to stop it). Fixed + by dropping the rook/bishop-file restriction entirely: ANY pawn on + the rank immediately before queening is now excluded, not just + rook/bishop pawns. + + After all three fixes: no more confidence left in reasoning our way + to a boundary here without re-deriving it from EGTB -- every "safe" + exception considered above (defended queen, non-rook/bishop file) + turned out to have a counterexample the first time it was actually + checked. Current scope is deliberately conservative as a result. + +Parameters: + + SEARCHER_THREAD_CONTEXT *ctx, + SCORE *piScore + +Return value: + + static ULONG + +**/ +{ + ULONG uStrong, uWeak; + COOR cPawn, cQueen, cWeakKing; + POSITION *pos = &ctx->sPosition; + + ASSERT((TotalNonPawnCount(pos, WHITE) <= 3) && + (TotalNonPawnCount(pos, BLACK) <= 3)); + ASSERT(_NothingBut(pos, PAWN | QUEEN, WHITE)); + ASSERT(_NothingBut(pos, PAWN | QUEEN, BLACK)); + + // + // Call the side with the queen "strong" + // + uStrong = WHITE; + if (TotalNonPawnCount(pos, BLACK) > 1) + { + uStrong = BLACK; + } + uWeak = FLIP(uStrong); + ASSERT(TotalNonPawnCount(pos, uStrong) > 1); + ASSERT(TotalNonPawnCount(pos, uWeak) == 1); + + // + // Only proven for exactly one queen vs exactly one pawn. + // + if ((FastCountBits(pos->bbPieces[uStrong][QUEEN]) != 1) || + (FastCountBits(pos->bbPawns[uWeak]) != 1)) + { + return(UNRECOGNIZED); + } + + // + // Queen safety: if the weak king can capture the queen, OR the weak + // pawn can capture the queen (necessarily while promoting, since + // the pawn has only one square left to go -- see this function's + // 2026-09-24 header comment for why that's catastrophic even when + // "defended"), this is not an unconditional win by any of the + // reasoning below. Bail regardless of whose move it actually is and + // regardless of whether the strong king defends the queen's square + // -- deliberately not trying to reason through defended-vs- + // undefended exceptions here again. + // + cQueen = CoorOfSingleBit(pos->bbPieces[uStrong][QUEEN]); + ASSERT(IS_ON_BOARD(cQueen)); + cWeakKing = KingPosition(pos, uWeak); + if ((DISTANCE(cQueen, cWeakKing) == 1) || + (0 != (g_PawnAttackOriginBB[uWeak][cQueen] & pos->bbPawns[uWeak]))) + { + return(UNRECOGNIZED); + } + + cPawn = CoorOfSingleBit(pos->bbPawns[uWeak]); + ASSERT(IS_ON_BOARD(cPawn)); + ASSERT(IS_PAWN(pos->rgSquare[cPawn].pPiece)); + + // + // Any pawn on the rank immediately before queening -- not just a + // rook/bishop pawn -- threatens to promote uncontested next move + // (or, if it's the weak side's own move right now, THIS move), + // producing a second queen and rebalancing material regardless of + // file. See this function's 2026-09-24 header comment: the file- + // restricted version of this check missed exactly this on a + // knight-file pawn (K1k5/Q7/8/8/8/8/1p6/8 b - -, Black's b2 pawn, + // queen too far away to stop ...b1=Q) as well as the original + // rook/bishop-file "fortress" reasoning it was written for. + // + if (((WHITE == uWeak) && (TRUE == RANK7(cPawn))) || + ((BLACK == uWeak) && (TRUE == RANK2(cPawn)))) + { + return(UNRECOGNIZED); + } + + // + // Every other pawn placement is an unconditional win for the queen + // side with best play. Material balance is already reliably very + // positive for the queen side here (a queen vastly outweighs a lone + // pawn), so it's safe to use directly as a directional bound -- we + // don't need (or attempt) a precise mate-distance score, just a + // correctly-signed one. + // + *piScore = pos->iMaterialBalance[uStrong]; + if (pos->uToMove == uWeak) + { + *piScore *= -1; + return(RECOGN_UPPER); + } + return(RECOGN_LOWER); +} + + static ULONG _RecognizeKBKP(IN SEARCHER_THREAD_CONTEXT *ctx, IN OUT SCORE *piScore) @@ -1912,6 +2075,27 @@ Return value: _MakeMaterialSig(1, 0, 0, 0, 0), _MakeMaterialSig(0, 1, 0, 0, 0)); + // KQ+KP+ DISABLED (like KBNK above): TestRecognExhaustiveKQKP found + // three structurally distinct correctness holes in a row, each only + // surfacing once the previous one was fixed -- queen safety vs. the + // weak king, queen safety vs. a promoting weak pawn, and any weak + // pawn (not just rook/bishop-file) reaching the queening-adjacent + // rank threatening to rebalance material via a second queen. After + // those three fixes, weak-side stalemate accounted for only 1,746 + // of the remaining 42,398 wrong positions (12.7M checked) -- the + // other 40,652 are a fourth, still-unidentified failure mode. Given + // that pattern, this "queen always beats a lone pawn" classical + // claim has more real exceptions across the full space of legal + // positions than can safely be hand-derived from memory one fix at + // a time. Not registered until someone re-derives the actual + // correct boundary (most plausibly by deriving it FROM Syzygy WDL + // data rather than guessing and re-testing) and re-validates via + // TestRecognExhaustiveKQKP. + // + // _NewRecognizer(_RecognizeKQKP, + // _MakeMaterialSig(1, 0, 0, 0, 0), + // _MakeMaterialSig(0, 0, 0, 0, 1)); + // KB+KP+ P N B R Q _NewRecognizer(_RecognizeKBKP, _MakeMaterialSig(1, 0, 0, 0, 0), @@ -1943,7 +2127,7 @@ Return value: ULONG RecognLookup(IN SEARCHER_THREAD_CONTEXT *ctx, IN OUT SCORE *piScore, - IN FLAG fProbeEGTB) + IN ULONG uEgtbProbeMode) /** Routine description: @@ -1954,11 +2138,23 @@ Routine description: present on the board and 2. probe on-disk EGTB files if no useful recognizer result is found. + uEgtbProbeMode is trinary (EGTB_PROBE_NEVER / EGTB_PROBE_ALWAYS / + EGTB_PROBE_IF_RESIDENT, chess.h) rather than a plain FLAG so that + "probe iff we believe the page is resident" can be decided in here, + on only the branch that's actually about to call ProbeEGTB, instead + of the caller computing EgtbProbeLikelyResident unconditionally as + a function argument before even knowing whether an interior + recognizer below is about to make the question moot. This matters + more than it used to: EgtbProbeLikelyResident can now involve real + mincore() syscalls for 6-man material (see its own comment in + probe.c) rather than just a cheap atomic load, so deferring it here + directly cuts how often that runs. + Parameters: SEARCHER_THREAD_CONTEXT *ctx, SCORE *piScore, - FLAG fProbeEGTB + ULONG uEgtbProbeMode Return value: @@ -1972,9 +2168,7 @@ Return value: RECOGNIZER *pRec; SCORE iScore; - // - // Try interior node recognizers - // + // Try interior node recognizers first. if ((TotalNonPawnCount(pos, WHITE) <= 3) && (TotalNonPawnCount(pos, BLACK) <= 3)) { @@ -2017,14 +2211,12 @@ Return value: } } #endif - ASSERT(_SanityCheckRecognizers(ctx, iScore, uVal)); return(uVal); } } } } - // // Try EGTB probe. No piece-count gate here: ProbeEGTB already // checks the position's piece count against the dynamic // TB_LARGEST (set from whatever tablebase files Fathom actually @@ -2034,17 +2226,27 @@ Return value: // ever added alongside the 5-man set already present -- see // CLAUDE.md). // - if (FALSE != fProbeEGTB) + // EGTB_PROBE_IF_RESIDENT's EgtbProbeLikelyResident call happens + // right here, not in the caller -- this is the whole point of + // uEgtbProbeMode being trinary instead of a plain FLAG (see this + // function's header comment): every node above that an interior + // recognizer already resolved returned before ever reaching this + // line, so it never pays for a residency check (which can mean + // real mincore() syscalls for 6-man material) it wouldn't use. + if (EGTB_PROBE_NEVER != uEgtbProbeMode) { - if (TRUE == ProbeEGTB(ctx, &iScore)) + FLAG fShouldProbe = (EGTB_PROBE_ALWAYS == uEgtbProbeMode) || + EgtbProbeLikelyResident(ctx); + if (TRUE == fShouldProbe) { - // - // Syzygy WDL probes are win/draw/loss only -- there's no - // mate distance to adjust for the present ply here (see - // EGTB_WIN_SCORE / EGTB_LOSS_SCORE in probe.c). - // - *piScore = iScore; - return(RECOGN_EXACT); + if (TRUE == ProbeEGTB(ctx, &iScore)) + { + // Syzygy WDL probes are win/draw/loss only -- there's no + // mate distance to adjust for the present ply here (see + // EGTB_WIN_SCORE / EGTB_LOSS_SCORE in probe.c). + *piScore = iScore; + return(RECOGN_EXACT); + } } } return(UNRECOGNIZED); @@ -501,46 +501,6 @@ Return value: Trace("Never used endgame pruning heuristic.\n"); } #endif -#ifdef TEST_NULL - n = (double)(ctx->sCounters.tree.u64AvoidNullSuccess); - d = n + (double)(ctx->sCounters.tree.u64AvoidNullFailures); - if (d != 0.0) - { - Trace("Avoid null move heuristic rate was %5.3f percent.\n" - "Avoid null move heuristic was " - "%"COMPILER_LONGLONG_UNSIGNED_FORMAT " / " - "%"COMPILER_LONGLONG_UNSIGNED_FORMAT ".\n", - ((n / d) * 100.0), - ctx->sCounters.tree.u64AvoidNullSuccess, - (ctx->sCounters.tree.u64AvoidNullSuccess + - ctx->sCounters.tree.u64AvoidNullFailures)); - } - else - { - Trace("Never used the avoid null move heuristic.\n"); - } - n = (double)(ctx->sCounters.tree.u64QuickNullSuccess); - n += (double)(ctx->sCounters.tree.u64QuickNullDeferredSuccess); - d = n + (double)(ctx->sCounters.tree.u64QuickNullFailures); - if (d != 0.0) - { - Trace("Quick null move heuristic rate was %5.3f percent.\n" - "Quick null move heuristic rate was " - "%"COMPILER_LONGLONG_UNSIGNED_FORMAT " (" - "%"COMPILER_LONGLONG_UNSIGNED_FORMAT ") / " - "%"COMPILER_LONGLONG_UNSIGNED_FORMAT ".\n", - ((n / d) * 100.0), - ctx->sCounters.tree.u64QuickNullSuccess, - ctx->sCounters.tree.u64QuickNullDeferredSuccess, - (ctx->sCounters.tree.u64QuickNullSuccess + - ctx->sCounters.tree.u64QuickNullDeferredSuccess + - ctx->sCounters.tree.u64QuickNullFailures)); - } - else - { - Trace("Never used the quick null move heuristic.\n"); - } -#endif n = (double)ctx->sCounters.tree.u64BetaCutoffsOnFirstMove; d = (double)ctx->sCounters.tree.u64BetaCutoffs + 1; Trace("First move beta cutoff rate was %5.3f percent.\n", @@ -1547,6 +1507,21 @@ Return value: g_Options.u64CMEvidenceTries[u] = ctx->sCounters.tree.u64CMEvidenceTries[u]; g_Options.u64CMEvidenceFH[u] = ctx->sCounters.tree.u64CMEvidenceFH[u]; } +#ifdef TEST_NULL + for (u = 0; u < NC_TOTAL_BUCKETS; u++) + { + g_Options.u64NCAttempts[u] = ctx->sCounters.tree.u64NCAttempts[u]; + g_Options.u64NCNullFH[u] = ctx->sCounters.tree.u64NCNullFH[u]; + g_Options.u64NCRealFH[u] = ctx->sCounters.tree.u64NCRealFH[u]; + g_Options.u64NCBothFH[u] = ctx->sCounters.tree.u64NCBothFH[u]; + g_Options.u64NCNullProbeNodeSum[u] = + ctx->sCounters.tree.u64NCNullProbeNodeSum[u]; + g_Options.u64NCFirstMoveNodeSum[u] = + ctx->sCounters.tree.u64NCFirstMoveNodeSum[u]; + g_Options.u64NCRealLoopNodeSum[u] = + ctx->sCounters.tree.u64NCRealLoopNodeSum[u]; + } +#endif g_MoveTimer.dEndTime = SystemTimeStamp(); // diff --git a/src/script.c b/src/script.c index 532a0d5..f18ab7b 100644 --- a/src/script.c +++ b/src/script.c @@ -64,6 +64,24 @@ typedef struct _SUITE_COUNTERS UINT64 u64TotalLeftoverFH; UINT64 u64TotalCMEvidenceTries[CM_EVIDENCE_BUCKETS]; UINT64 u64TotalCMEvidenceFH[CM_EVIDENCE_BUCKETS]; +#ifdef TEST_NULL + // Null-move precision/recall calibration, summed across every + // position in the suite -- see _NullCalibBucketIndex's comment in + // search.c. Accumulated the same way as u64TotalCMEvidenceTries + // above (per-search counters are copied into g_Options at the end + // of each Think() call, then summed in here by _ProcessSuiteLine), + // and printed once by _PrintSuiteReport at true end-of-suite -- + // deliberately *not* dumped after every position, which would + // balloon the logfile (confirmed: 191MB after ~28K of 200K + // positions the first time this was tried that way). + UINT64 u64TotalNCAttempts[NC_TOTAL_BUCKETS]; + UINT64 u64TotalNCNullFH[NC_TOTAL_BUCKETS]; + UINT64 u64TotalNCRealFH[NC_TOTAL_BUCKETS]; + UINT64 u64TotalNCBothFH[NC_TOTAL_BUCKETS]; + UINT64 u64TotalNCNullProbeNodeSum[NC_TOTAL_BUCKETS]; + UINT64 u64TotalNCFirstMoveNodeSum[NC_TOTAL_BUCKETS]; + UINT64 u64TotalNCRealLoopNodeSum[NC_TOTAL_BUCKETS]; +#endif double dSigmaEBF; // sum of per-problem nodes^(1/depth) ULONG uEBFCount; // number of problems with depth > 0 double dAverageTimeToSolution; @@ -391,6 +409,28 @@ _ProcessSuiteLine(char *szLine, POSITION *pos) g_Options.u64CMEvidenceFH[v]; } } +#ifdef TEST_NULL + { + ULONG v; + for (v = 0; v < NC_TOTAL_BUCKETS; v++) + { + g_SuiteCounters.u64TotalNCAttempts[v] += + g_Options.u64NCAttempts[v]; + g_SuiteCounters.u64TotalNCNullFH[v] += + g_Options.u64NCNullFH[v]; + g_SuiteCounters.u64TotalNCRealFH[v] += + g_Options.u64NCRealFH[v]; + g_SuiteCounters.u64TotalNCBothFH[v] += + g_Options.u64NCBothFH[v]; + g_SuiteCounters.u64TotalNCNullProbeNodeSum[v] += + g_Options.u64NCNullProbeNodeSum[v]; + g_SuiteCounters.u64TotalNCFirstMoveNodeSum[v] += + g_Options.u64NCFirstMoveNodeSum[v]; + g_SuiteCounters.u64TotalNCRealLoopNodeSum[v] += + g_Options.u64NCRealLoopNodeSum[v]; + } + } +#endif } else { @@ -473,6 +513,103 @@ _PrintSuiteReport(double dSuiteStart) } } +#ifdef TEST_NULL + // Null-move precision/recall calibration, summed across the whole + // suite (see _NullCalibBucketIndex's comment in search.c for the + // encoding and what each dimension means). Decode each bucket + // index back into its component bands for a readable line; skip + // empty cells so this stays legible even though the full table has + // NC_TOTAL_BUCKETS cells. + { + static const char *rgszDepthBand[NC_DEPTH_BUCKETS] = + { "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", + "11", "12", "13", "14", "15", "16+" }; + static const char *rgszDeficitBand[NC_DEFICIT_BUCKETS] = + { "<-300", "-300..-100", "-100..0", "0..100", "100..300", "300+" }; + static const char *rgszMaterialBand[NC_MATERIAL_BUCKETS] = + { "0-1", "2", "3-5", "6+" }; + ULONG uIdx, uDecode, uNullDepthB, uOuterDepthB, uDeficitB; + ULONG uMaxMaterialB, uMoverMaterialB; + UINT64 u64Attempts, u64NullFH, u64RealFH, u64BothFH; + UINT64 u64NullProbeNodeSum, u64FirstMoveNodeSum, u64RealLoopNodeSum; + double dAvgNullProbeNodes, dAvgFirstMoveNodes, dAvgRealLoopNodes; + double dNullSuccessRate, dEVFirstMoveOnly, dEV; + + Trace("\nNull-move calibration (attempts / nullFH / realFH / both / " + "precision / recall / nullSuccessRate / avgNullProbeNodes / " + "avgFirstMoveNodes / avgRealLoopNodes / EVFirstMoveOnly / " + "EV-per-attempt-in-nodes):\n"); + for (uIdx = 0; uIdx < NC_TOTAL_BUCKETS; uIdx++) + { + u64Attempts = g_SuiteCounters.u64TotalNCAttempts[uIdx]; + if (0 == u64Attempts) continue; + u64NullFH = g_SuiteCounters.u64TotalNCNullFH[uIdx]; + u64RealFH = g_SuiteCounters.u64TotalNCRealFH[uIdx]; + u64BothFH = g_SuiteCounters.u64TotalNCBothFH[uIdx]; + u64NullProbeNodeSum = g_SuiteCounters.u64TotalNCNullProbeNodeSum[uIdx]; + u64FirstMoveNodeSum = g_SuiteCounters.u64TotalNCFirstMoveNodeSum[uIdx]; + u64RealLoopNodeSum = g_SuiteCounters.u64TotalNCRealLoopNodeSum[uIdx]; + + // Decode uIdx (innermost/fastest-varying dimension first), + // matching _NullCalibBucketIndex's encoding in search.c. + // Use a scratch copy -- uIdx itself is still the array + // index for the lookups above/below. + uDecode = uIdx; + uMoverMaterialB = uDecode % NC_MATERIAL_BUCKETS; + uDecode /= NC_MATERIAL_BUCKETS; + uMaxMaterialB = uDecode % NC_MATERIAL_BUCKETS; + uDecode /= NC_MATERIAL_BUCKETS; + uDeficitB = uDecode % NC_DEFICIT_BUCKETS; + uDecode /= NC_DEFICIT_BUCKETS; + uOuterDepthB = uDecode % NC_DEPTH_BUCKETS; + uDecode /= NC_DEPTH_BUCKETS; + uNullDepthB = uDecode % NC_DEPTH_BUCKETS; + + dAvgNullProbeNodes = (double)u64NullProbeNodeSum / (double)u64Attempts; + dAvgFirstMoveNodes = (double)u64FirstMoveNodeSum / (double)u64Attempts; + dAvgRealLoopNodes = (double)u64RealLoopNodeSum / (double)u64Attempts; + dNullSuccessRate = (double)u64NullFH / (double)u64Attempts; + // EV of TRYING null vs. going straight to the real move loop, + // in nodes saved per attempt (positive = trying pays off): + // on the dNullSuccessRate fraction of attempts we pay the + // (cheap) null probe instead of the (expensive) real-loop + // cost; on the rest we pay the null probe *in addition* to + // still needing the real loop. dEV uses avgRealLoopNodes -- + // the true total cost of however the real search actually + // resolved the node (one move, several, or production's own + // separate null success) -- while dEVFirstMoveOnly keeps the + // cheaper move-1-only proxy alongside it for comparison. + // Both assume (per the user's simplifying assumption, + // matched by this dataset's ~99.96% overall precision) that + // a null-move fail-high is correct. + dEVFirstMoveOnly = + dNullSuccessRate * (dAvgFirstMoveNodes - dAvgNullProbeNodes) + - (1.0 - dNullSuccessRate) * dAvgNullProbeNodes; + dEV = dNullSuccessRate * (dAvgRealLoopNodes - dAvgNullProbeNodes) + - (1.0 - dNullSuccessRate) * dAvgNullProbeNodes; + + Trace(" nullDepth=%s outerDepth=%s deficit=%s " + "maxMaterial=%s moverMaterial=%s : " + "%"COMPILER_LONGLONG_UNSIGNED_FORMAT" / " + "%"COMPILER_LONGLONG_UNSIGNED_FORMAT" / " + "%"COMPILER_LONGLONG_UNSIGNED_FORMAT" / " + "%"COMPILER_LONGLONG_UNSIGNED_FORMAT" / " + "%5.3f / %5.3f / %5.3f / %10.1f / %10.1f / %10.1f / %10.1f / %10.1f\n", + rgszDepthBand[uNullDepthB], + rgszDepthBand[uOuterDepthB], + rgszDeficitBand[uDeficitB], + rgszMaterialBand[uMaxMaterialB], + rgszMaterialBand[uMoverMaterialB], + u64Attempts, u64NullFH, u64RealFH, u64BothFH, + (u64NullFH != 0) ? (100.0 * (double)u64BothFH / (double)u64NullFH) : 0.0, + (u64RealFH != 0) ? (100.0 * (double)u64BothFH / (double)u64RealFH) : 0.0, + 100.0 * dNullSuccessRate, + dAvgNullProbeNodes, dAvgFirstMoveNodes, dAvgRealLoopNodes, + dEVFirstMoveOnly, dEV); + } + } +#endif + // Histogram stuff if (g_SuiteCounters.uTotal > 0) { uMax = g_SuiteCounters.uHistogram[0]; diff --git a/src/search.c b/src/search.c index 0362701..8f4779e 100644 --- a/src/search.c +++ b/src/search.c @@ -104,6 +104,131 @@ _CMEvidenceBucket(ULONG uEvidence) } #endif +#ifdef TEST_NULL +// +// Null-move try/avoid calibration harness. +// +// Question: given the current data (precision >99.9% almost everywhere, +// false-positive/"null-move lie" risk concentrated by material and +// deficit, EV strongly positive almost everywhere once restricted to +// the safe region), can a two-stage try/skip policy do better than +// today's flat material threshold + per-depth eval-margin table? +// +// Stage 1 (safety): is a null-move fail-high here even trustworthy? +// Stage 2 (profitability/EV): given it's safe, is it worth the cost? +// +// fThreat, fSkipNull, and en-prise/trapped-piece state were bucketed in +// an earlier revision of this harness (2026-09-24 run, saved under +// nullmove_calib/run_2026-09-24_sd10_30k/) and shown to be either +// already-validated non-tunable binary gates (fThreat/fSkipNull -- both +// clearly reduce EV and precision/recall when set, and both already +// gate the call to WeShouldTryNullmovePruning in Search(), so there's +// no threshold left to calibrate) or not a safety signal at all +// (en-prise/trapped predicts *lower recall*, not higher false-positive +// risk -- a Stage 2, not Stage 1, concern). None of the three are +// bucketed here; fThreat==FALSE && fSkipNull==FALSE are folded into +// this probe's structural eligibility instead, matching production. +// +// This revision instead resolves two open questions from that first +// run: +// +// 1. Is MAX(white,black) non-pawn piece count or the MOVER's own +// count (TotalNonPawnCount(pos, pos->uToMove)) the better +// Stage-1 safety predictor? Zugzwang is a property of the side +// being given the free "pass", so mover-only is the more +// causally-motivated signal, but the first run only measured +// MAX(both) -- bucket both, jointly, and let the false-positive- +// rate data settle it. +// 2. The first run bucketed the *real* search's cost (the Stage-2 EV +// calc's "cost of the alternative") by the null probe's own post- +// reduction depth (uNullDepth), which is only meaningful via +// whatever R schedule (SelectNullmoveRFactor) happened to be +// compiled in at the time -- useless for evaluating a *different* +// R schedule later. The real search's cost is fundamentally a +// function of the OUTER, pre-reduction uDepth (how much real work +// remains), not of uNullDepth at all. Bucket both depths +// separately: uNullDepth for the probe's own cost (cost1), uDepth +// for the real alternative's cost (cost2) -- decoupling the two +// cost models from any particular R formula. +// +// Ground truth (did this node *really* fail high) is read off the same +// iBestScore-vs-iBeta comparison Search() already computes at its +// "end:" label for the NODE_PV/NODE_CUT/NODE_ALL surprise counters just +// above this block; we piggyback on that rather than duplicating it. +// The diagnostic probe is a self-contained MakeMove(NULLMOVE) -> reduced +// Search() -> UnmakeMove, deliberately *not* reusing TryNullmovePruning, +// so its verification-search/extension side effects (meant only for a +// probe whose result production actually trusts) never fire here. Like +// any double-search diagnostic, it also warms the hash/history tables +// along its own subtree before the real move loop runs at the same +// node -- a known, unavoidable side effect of measuring this way. +// +// This roughly doubles node cost at every non-PV node under TEST_NULL, +// so it's a diagnostic-only build flag (gmake TEST_NULL=1 ...), never +// part of the normal GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1 profile, +// and should only be run against a calibration corpus at a shallow +// fixed depth, not self-play or the full engine pools. +// +// Buckets are a mixed-radix index over five dimensions: +// +// index = nullDepthB * (NC_DEPTH_BUCKETS*NC_DEFICIT_BUCKETS*NC_MATERIAL_BUCKETS*NC_MATERIAL_BUCKETS) +// + outerDepthB * (NC_DEFICIT_BUCKETS*NC_MATERIAL_BUCKETS*NC_MATERIAL_BUCKETS) +// + deficitB * (NC_MATERIAL_BUCKETS*NC_MATERIAL_BUCKETS) +// + maxMaterialB* (NC_MATERIAL_BUCKETS) +// + moverMaterialB +// +// root.c's dump loop decodes this same layout when printing results. +// +static ULONG +_NullCalibDepthBucket(ULONG uSomeDepth) // generic: used for both uNullDepth and uDepth +{ + ULONG uPlies = uSomeDepth / ONE_PLY; + if (uPlies >= NC_DEPTH_BUCKETS - 1) return(NC_DEPTH_BUCKETS - 1); // 16+ + return(uPlies); +} + +static ULONG +_NullCalibDeficitBucket(SCORE iDeficit) // iEval - iBeta +{ + if (iDeficit < -300) return(0); + if (iDeficit < -100) return(1); + if (iDeficit < 0) return(2); + if (iDeficit < 100) return(3); + if (iDeficit < 300) return(4); + return(5); +} + +static ULONG +_NullCalibMaterialBucket(ULONG uPiecesPerSide) // generic: used for both MAX(both) and mover-only +{ + if (uPiecesPerSide <= 1) return(0); + if (uPiecesPerSide == 2) return(1); + if (uPiecesPerSide <= 5) return(2); + return(3); +} + +static ULONG +_NullCalibBucketIndex(ULONG uNullDepth, // probe's own post-reduction depth (cost1 axis) + ULONG uOuterDepth, // real node's pre-reduction depth (cost2 axis) + SCORE iDeficit, + ULONG uMaxPiecesPerSide, // MAX(white,black) + ULONG uMoverPiecesPerSide) // TotalNonPawnCount(pos, pos->uToMove) +{ + ULONG uNullDepthB = _NullCalibDepthBucket(uNullDepth); + ULONG uOuterDepthB = _NullCalibDepthBucket(uOuterDepth); + ULONG uDeficitB = _NullCalibDeficitBucket(iDeficit); + ULONG uMaxMaterialB = _NullCalibMaterialBucket(uMaxPiecesPerSide); + ULONG uMoverMaterialB = _NullCalibMaterialBucket(uMoverPiecesPerSide); + ULONG uIndex = uNullDepthB; + uIndex = uIndex * NC_DEPTH_BUCKETS + uOuterDepthB; + uIndex = uIndex * NC_DEFICIT_BUCKETS + uDeficitB; + uIndex = uIndex * NC_MATERIAL_BUCKETS + uMaxMaterialB; + uIndex = uIndex * NC_MATERIAL_BUCKETS + uMoverMaterialB; + ASSERT(uIndex < NC_TOTAL_BUCKETS); + return(uIndex); +} +#endif + #ifdef DEBUG #define VERIFY_HASH_HIT \ ASSERT(IS_VALID_SCORE(iScore)); \ @@ -190,6 +315,15 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx, FLAG fAnyMoveEFPPruned = FALSE; FLAG fThisMoveEFPPruned = FALSE; SCORE iMoveSee; +#ifdef TEST_NULL + FLAG fNullCalibEligible = FALSE; + FLAG fNullCalibProbeFH = FALSE; + ULONG uNullCalibBucket = 0; + UINT64 uNullCalibProbeNodeCost = 0; + UINT64 uNullCalibFirstMoveNodeCost = 0; + UINT64 uNullCalibRealLoopNodeCost = 0; + UINT64 uNullCalibNodesBeforeRealLoop = 0; +#endif #ifdef DEBUG ASSERT(IS_VALID_SCORE(iAlpha)); ASSERT(IS_VALID_SCORE(iBeta)); @@ -320,8 +454,18 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx, } // Probe interior node recognizers; allow probes of ondisk EGTB files - // if it looks like we can get hit. - switch(RecognLookup(ctx, &iScore, ctx->uPly <= (g_uIterateDepth / 2))) + // if we're shallow (near the root, where a first-touch mmap()/disk + // read is affordable) -- always probe there -- *or*, deeper, only + // if this exact position's lookup is likely resident (mincore()- + // gated for 6-man material; a handful of popcounts and one atomic + // load for 3-5 man -- see probe.c's EgtbProbeLikelyResident and + // CLAUDE.md). That residency check itself is deferred into + // RecognLookup (EGTB_PROBE_IF_RESIDENT), not computed here, so it's + // skipped entirely at any node a cheap interior recognizer already + // resolves -- see RecognLookup's own header comment. + switch(RecognLookup(ctx, &iScore, + (ctx->uPly <= (g_uIterateDepth / 2)) ? + EGTB_PROBE_ALWAYS : EGTB_PROBE_IF_RESIDENT)) { case UNRECOGNIZED: break; @@ -362,6 +506,83 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx, } GENERATE_NO_MOVES; + +#ifdef TEST_NULL + // See _NullCalibBucketIndex's comment above for the full design. + // Structural eligibility only -- deliberately does *not* consult + // fSkipNull/fThreat/WeShouldTryNullmovePruning's margin table or + // material threshold, since those are exactly the heuristics this + // harness exists to evaluate. Note: like any double-search + // diagnostic, this probe's recursive Search() call warms the hash + // and history/countermove tables along its own subtree before the + // real move loop below runs at this same node -- a known, + // unavoidable side effect of measuring this way, not a bug. + // + // MUST run after GENERATE_NO_MOVES, not before: that macro is what + // sets ctx->sMoveStack.uBegin[ctx->uPly+1] (from uEnd[ctx->uPly]), + // and recursing into ctx->uPly+1 before it runs uses a stale + // leftover index from whatever unrelated node last occupied that + // ply slot -- crashed hard (MVF_MOVE_SEARCHED-already-set asserts, + // corrupting a real, in-flight move a few frames up) the first time + // this was tried with the block placed above GENERATE_NO_MOVES. + fNullCalibEligible = ((FALSE == pf->fAvoidNullmove) && + (FALSE == pi->fInCheck) && + (iBeta != +INFINITY_SCORE) && + (NODE_PV != pi->eNodeType) && + (FALSE == fThreat) && + (FALSE == fSkipNull)); + if (fNullCalibEligible) + { + ULONG uMaxPiecesPerSide = MAXU(TotalNonPawnCount(pos, WHITE), + TotalNonPawnCount(pos, BLACK)); + ULONG uMoverPiecesPerSide = TotalNonPawnCount(pos, pos->uToMove); + SCORE iNullCalibScore; + UINT64 u64NodesBefore; + + uNullCalibBucket = _NullCalibBucketIndex(uNextDepth, + uDepth, + (SCORE)(iEval - iBeta), + uMaxPiecesPerSide, + uMoverPiecesPerSide); + VERIFY(MakeMove(ctx, NULLMOVE)); + pf->fAvoidNullmove = TRUE; + u64NodesBefore = ctx->sCounters.tree.u64TotalNodeCount; + iNullCalibScore = -Search(ctx, -iBeta, -iBeta + 1, uNextDepth); + uNullCalibProbeNodeCost = ctx->sCounters.tree.u64TotalNodeCount - u64NodesBefore; + pf->fAvoidNullmove = FALSE; + UnmakeMove(ctx, NULLMOVE); + + if (!WE_SHOULD_STOP_SEARCHING) + { + // Direct increments, not INC() -- these calibration counters + // must not depend on PERF_COUNTERS also being defined; a + // TEST_NULL=1 build without PERF_COUNTERS=1 would otherwise + // silently collect nothing (INC() is a no-op without it). + fNullCalibProbeFH = (iNullCalibScore >= iBeta); + ctx->sCounters.tree.u64NCAttempts[uNullCalibBucket]++; + ctx->sCounters.tree.u64NCNullProbeNodeSum[uNullCalibBucket] += + uNullCalibProbeNodeCost; + if (fNullCalibProbeFH) + { + ctx->sCounters.tree.u64NCNullFH[uNullCalibBucket]++; + } + } + else + { + fNullCalibEligible = FALSE; // don't commit ground truth either + } + // Snapshot right here, before production's own null-move logic + // and the real move loop below -- the delta read at "end:" is + // the true total cost of however this node actually gets + // resolved from this point on (whether that's production's own + // separate null success, one real move, or several). + if (fNullCalibEligible) + { + uNullCalibNodesBeforeRealLoop = ctx->sCounters.tree.u64TotalNodeCount; + } + } +#endif + if (!fSkipNull && !fThreat && WeShouldTryNullmovePruning(ctx, @@ -925,6 +1146,24 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx, { // First move, full a..b window. ASSERT(uLegalMoves == 1); +#ifdef TEST_NULL + // Cost side of the null-move calibration harness' + // try-vs-skip EV estimate: this is the "R=0" + // alternative the null probe (see above) is being + // weighed against -- the real first move's own + // full-depth subtree cost. uLegalMoves==1 guarantees + // this fires exactly once per node. + if (fNullCalibEligible) + { + UINT64 u64NodesBeforeFirstMove = + ctx->sCounters.tree.u64TotalNodeCount; + iScore = -Search(ctx, -iBeta, -iAlpha, uNextDepth); + uNullCalibFirstMoveNodeCost = + ctx->sCounters.tree.u64TotalNodeCount - + u64NodesBeforeFirstMove; + } + else +#endif iScore = -Search(ctx, -iBeta, -iAlpha, uNextDepth); } else @@ -1162,6 +1401,58 @@ Search(IN SEARCHER_THREAD_CONTEXT *ctx, } } #endif +#ifdef TEST_NULL + // Commit the calibration record stashed at the diagnostic probe + // site above, now that iBestScore's fate against iBeta (this + // node's real, ground-truth fail-high outcome) is known. + if (fNullCalibEligible && + (!WE_SHOULD_STOP_SEARCHING) && + IS_VALID_SCORE(iBestScore)) + { + // Direct increments -- see the probe site's comment on why + // these can't use INC(). + FLAG fRealFH = (iBestScore >= iBeta); + if (fRealFH) + { + ctx->sCounters.tree.u64NCRealFH[uNullCalibBucket]++; + if (fNullCalibProbeFH) + { + ctx->sCounters.tree.u64NCBothFH[uNullCalibBucket]++; + } + } + else if (fNullCalibProbeFH) + { + // False positive: the null-move probe predicted a fail-high + // that the real move loop didn't confirm -- exactly the + // "null-move lie" case that a low-material verification + // search exists to catch in production. Dump a capped + // number of example FENs per bucket (same "capped FEN + // sampler" idea as eval.c's MARGINSAFETY_EXCEEDED logging) + // so these can be inspected by hand for genuine zugzwang + // our TotalNonPawnCount/ValueOfMaterialInTroubleAfterNull + // safety net is missing, vs. ordinary search noise. + static ULONG uFalsePositiveFenSamples[NC_TOTAL_BUCKETS]; + if (uFalsePositiveFenSamples[uNullCalibBucket] < 5) + { + CHAR *pszFen = PositionToFen(pos); + if (NULL != pszFen) + { + uFalsePositiveFenSamples[uNullCalibBucket]++; + Trace("NULLCALIB_FALSE_POSITIVE bucket=%lu depth=%lu " + "fen=%s\n", + uNullCalibBucket, uDepth, pszFen); + SystemFreeMemory(pszFen); + } + } + } + ctx->sCounters.tree.u64NCFirstMoveNodeSum[uNullCalibBucket] += + uNullCalibFirstMoveNodeCost; + uNullCalibRealLoopNodeCost = ctx->sCounters.tree.u64TotalNodeCount - + uNullCalibNodesBeforeRealLoop; + ctx->sCounters.tree.u64NCRealLoopNodeSum[uNullCalibBucket] += + uNullCalibRealLoopNodeCost; + } +#endif DTLeaveNode(ctx, FALSE, iBestScore, mvBest); return(iBestScore); } @@ -1994,9 +2285,17 @@ QSearch(IN SEARCHER_THREAD_CONTEXT *ctx, } DTEnterNode(ctx, 0, TRUE, iAlpha, iBeta); - // Probe interior node recognizers; do not allow probes into ondisk - // EGTB files since we are in qsearch. - switch(RecognLookup(ctx, &iScore, FALSE)) + // Probe interior node recognizers. Qsearch runs at the deepest, + // most performance-sensitive part of the tree, so this used to + // unconditionally refuse EGTB probes -- but a qsearch node with few + // enough pieces to be in the tablebase at all is exactly the + // "trades have happened near a leaf" case the EGTB is most useful + // for: only probe when this position's lookup is likely resident + // (EGTB_PROBE_IF_RESIDENT -- RecognLookup itself decides, via + // EgtbProbeLikelyResident, only on the branch that's actually about + // to call ProbeEGTB, so a node an interior recognizer resolves + // never pays for the residency check at all) -- see CLAUDE.md. + switch(RecognLookup(ctx, &iScore, EGTB_PROBE_IF_RESIDENT)) { case UNRECOGNIZED: break; diff --git a/src/searchsup.c b/src/searchsup.c index 781c702..0201219 100644 --- a/src/searchsup.c +++ b/src/searchsup.c @@ -1382,17 +1382,26 @@ ULONG SelectNullmoveRFactor(SEARCHER_THREAD_CONTEXT *ctx, INT uDepth) { - ULONG u = TWO_PLY; + ULONG u; ULONG uMaxPiecesPerSide = MAXU(TotalNonPawnCount(&ctx->sPosition, WHITE), TotalNonPawnCount(&ctx->sPosition, BLACK)); ASSERT(uMaxPiecesPerSide <= 16); - if ((uDepth > 8 * ONE_PLY) || - ((uDepth > 6 * ONE_PLY) && (uMaxPiecesPerSide >= 3))) + if (uMaxPiecesPerSide < 3) { - u = THREE_PLY; + // Sparse material on the mover's side: stay with the + // conservative, non-growing reduction rather than risk a + // zugzwang-prone position getting an aggressive R. (The + // TotalNonPawnCount < 4 verification search in + // TryNullmovePruning is the other half of this safety net.) + u = TWO_PLY; + } + else + { + // Crafty-style growth: R = 3 + depth / 6 (in whole plies), + // scaled into our fractional-ply depth units. + u = THREE_PLY + ((uDepth / (6 * ONE_PLY)) * ONE_PLY); } - ASSERT((u == TWO_PLY) || (u == THREE_PLY)); return(u); } @@ -1403,14 +1412,82 @@ FLAG WeShouldTryNullmovePruning(SEARCHER_THREAD_CONTEXT *ctx, SCORE iEval, SCORE iImprovement, ULONG uNullDepth) +/** + +Routine description: + + Two-stage try/skip decision, replacing the old flat >2-non-pawn- + piece gate plus per-depth eval-margin table. Calibrated from a + 30K-position, sd-10, ~9-billion-attempt null-move precision/recall/ + EV harness (2026-09-24, two revisions -- see + nullmove_calib/run_2026-09-24_sd10_30k_v2/, the second and + authoritative one). iImprovement and uNullDepth are currently + unused: the Stage 1 thresholds below are flat, not depth- or + improvement-scaled, and Stage 2 doesn't need its own gate at all + (see below). Kept in the signature to avoid touching call sites. + + Stage 1 (safety): is a null-move fail-high here even trustworthy? + Gated on the MOVER's own non-pawn piece count -- TotalNonPawnCount + (pos, pos->uToMove), NOT MAX(white,black) as an earlier revision of + this gate (and of the calibration harness that measured it) used. + Zugzwang is a property of the side being given the free "pass", so + the opponent's material shouldn't matter -- confirmed empirically: + (maxMaterial=6+, moverMaterial=2) measured a 0.30% false-positive + rate, indistinguishable from the worst low-material cells overall, + despite "6+" sounding safe. A materially-heavier opponent does not + protect a mover that is itself thin on material. + + Measured false-positive rate (a null fail-high the real search + didn't confirm) by mover's own material x eval-vs-beta deficit: + + deficit<-300 -300..-100 -100..0 0..100 100..300 300+ + mover 0-1 pcs: 5.57% 1.35% 0.82% 1.02% 0.28% 0.12% + mover 2 pcs: 3.73% 0.19% 0.31% 0.32% 0.02% 0.01% + mover 3-5 pcs: 0.91% 0.20% 0.09% 0.002% 0.001% 0.0005% + mover 6+ pcs: 0.02% 0.002% 0.001% 0.0002% 0.0001% 0.0001% + + 6+ pieces is safe (<=0.02%) at EVERY deficit measured, even the most + negative -- no deficit floor needed there at all. 2 pieces is + actually salvageable (0.01-0.02%, comparable to the safest cells + overall) once deficit >= 100, unlike a flatly-disabled <=2 rule + would assume -- but <=1 piece never gets safe (worst case 0.12% at + its *best* deficit band), so that tier is disabled outright rather + than deficit-gated. + + Stage 2 (profitability/EV): given Stage 1 passes, is trying still + worth the reduced-depth probe's node cost? Checked directly against + the same run's EV data (EV = successRate*(realLoopCost-nullProbeCost) + - (1-successRate)*nullProbeCost, in nodes saved per attempt) across + every (depth, material) cell that survives the Stage 1 gate above: + EV was strongly positive everywhere measured, from low single digits + at shallow depth up to tens of thousands of nodes at deep probes, + never negative. Stage 1 is already doing essentially all the + discriminating work, so Stage 2 currently just returns TRUE + unconditionally once Stage 1 passes -- no separate EV threshold is + justified by this data. Revisit if a future, deeper or wider + calibration run finds a safe-but-unprofitable pocket this one + didn't sample. + +Parameters: + + SEARCHER_THREAD_CONTEXT *ctx, + SCORE iAlpha, + SCORE iBeta, + SCORE iEval, + SCORE iImprovement, + ULONG uNullDepth + +Return value: + + FLAG + +**/ { - static SCORE iSkipShallowNullIfThisFarBelowAlpha[] = { - 700, 950, 1110, 1150, 1190, 1230, 1270, 0 - }; POSITION *pos = &ctx->sPosition; PLY_INFO *pi = &ctx->sPlyInfo[ctx->uPly]; CUMULATIVE_SEARCH_FLAGS *pf = &ctx->sSearchFlags; - ULONG u; + ULONG uMoverPiecesPerSide; + SCORE iDeficit; ASSERT(IS_VALID_SCORE(iAlpha) && IS_VALID_SCORE(iBeta)); ASSERT(iAlpha < iBeta); @@ -1418,25 +1495,32 @@ FLAG WeShouldTryNullmovePruning(SEARCHER_THREAD_CONTEXT *ctx, ASSERT(uNullDepth < MAX_PLY_PER_SEARCH * ONE_PLY); if ((FALSE == pf->fAvoidNullmove) && - (TotalNonPawnCount(pos, pos->uToMove) > 2) && (FALSE == pi->fInCheck) && (iBeta != +INFINITY_SCORE) && (NODE_PV != pi->eNodeType)) { - if (uNullDepth <= 6 * ONE_PLY) + uMoverPiecesPerSide = TotalNonPawnCount(pos, pos->uToMove); + iDeficit = iEval - iBeta; + + // ---- Stage 1: safety ---- + if (uMoverPiecesPerSide <= 1) { - u = uNullDepth / ONE_PLY; - ASSERT(u <= 6); - SCORE iMargin = iSkipShallowNullIfThisFarBelowAlpha[u]; - iMargin -= iImprovement; // Note: if improving, makes null more likely - iMargin = MAX(0, iMargin); // if not, less likely. - if ((iEval + iMargin <= iAlpha) || - ((iEval + iMargin / 2 <= iAlpha) && - (ValueOfMaterialInTroubleAfterNull(ctx, pos->uToMove)))) - { - return FALSE; - } + return FALSE; // never measured safe, even at its best deficit + } + if ((uMoverPiecesPerSide == 2) && (iDeficit < 100)) + { + return FALSE; + } + if ((uMoverPiecesPerSide <= 5) && (iDeficit < -100)) + { + return FALSE; } + // 6+ pieces: no deficit floor -- safe everywhere measured. + + // ---- Stage 2: efficiency/EV ---- + // EV is strongly positive at every depth/material cell that + // survives Stage 1 (see routine comment) -- nothing further to + // gate on here today. return TRUE; } return FALSE; diff --git a/src/testrecogn.c b/src/testrecogn.c index b64643e..10ee889 100644 --- a/src/testrecogn.c +++ b/src/testrecogn.c @@ -295,7 +295,7 @@ _TestRecognFenCases(IN OUT SEARCHER_THREAD_CONTEXT *ctx, IN OUT POSITION *pos) } ReInitializeSearcherContext(pos, ctx); - uVal = RecognLookup(ctx, &iScore, FALSE); + uVal = RecognLookup(ctx, &iScore, EGTB_PROBE_NEVER); if ((UNRECOGNIZED != uVal) && ((iScore <= -NMATE) || (iScore >= NMATE))) { @@ -387,7 +387,7 @@ Return value: uPawnB, uKnightB, uBishopB); ReInitializeSearcherContext(&pos, ctx); - uVal = RecognLookup(ctx, &iScore, FALSE); + uVal = RecognLookup(ctx, &iScore, EGTB_PROBE_NEVER); if (UNRECOGNIZED == uVal) { uUnrecognized++; @@ -624,6 +624,267 @@ Return value: } // +// Build a KQKP (one queen, one pawn) position directly from square +// placements. fBlackHasQueen selects which color plays the queen side +// (cStrongKing/cWeakKing name roles, not colors, in both cases) -- see +// TestRecognExhaustiveKQKP's header comment for why both assignments +// are enumerated explicitly rather than assumed symmetric. +// +static FLAG +_BuildKQKPPosition(OUT POSITION *pos, + IN COOR cStrongKing, IN COOR cWeakKing, + IN COOR cQueen, IN COOR cPawn, + IN ULONG uToMove, IN FLAG fBlackHasQueen) +{ + PIECE pBoard[128]; + CHAR szFen[256]; + CHAR *p; + ULONG x; + + memset(pBoard, 0, sizeof(pBoard)); + if (fBlackHasQueen) + { + pBoard[cStrongKing] = BLACK_KING; + pBoard[cWeakKing] = WHITE_KING; + pBoard[cQueen] = BLACK_QUEEN; + pBoard[cPawn] = WHITE_PAWN; + } + else + { + pBoard[cStrongKing] = WHITE_KING; + pBoard[cWeakKing] = BLACK_KING; + pBoard[cQueen] = WHITE_QUEEN; + pBoard[cPawn] = BLACK_PAWN; + } + + memset(szFen, 0, sizeof(szFen)); + p = szFen; + for (x = 0; x < 120; x++) + { + if ((x % 15 == 0) && (x != 0)) + { + *p++ = '/'; + } + if (!IS_ON_BOARD(x)) continue; + if (0 == pBoard[x]) + { + *p++ = '1'; + } + else + { + *p++ = *(PieceAbbrev(pBoard[x]) + 1); + } + } + *p++ = ' '; + *p++ = (WHITE == uToMove) ? 'w' : 'b'; + strcat(p, " - - 0 1"); + + if (FALSE == FenToPosition(pos, szFen)) + { + return(FALSE); + } + if (InCheck(pos, FLIP(pos->uToMove))) + { + return(FALSE); + } + return(TRUE); +} + +// +// Does the side to move in ctx's current position have at least one +// legal move? Used only to classify a mismatch as "the to-move side is +// simply stalemated" vs. something else -- not part of the recognizer +// itself. Mirrors root.c's _DetectPreMoveTerminalStates legal-move +// count idiom (GenerateMoves then MakeMove/UnmakeMove-filter), just +// short-circuiting on the first legal move found instead of counting +// them all. +// +static FLAG +_ToMoveHasLegalMove(IN OUT SEARCHER_THREAD_CONTEXT *ctx) +{ + ULONG x; + MOVE mv; + FLAG fInCheck = InCheck(&ctx->sPosition, ctx->sPosition.uToMove); + + GenerateMoves(ctx, NULLMOVE, + fInCheck ? GENERATE_ESCAPES : GENERATE_ALL_MOVES); + for (x = ctx->sMoveStack.uBegin[ctx->uPly]; + x < ctx->sMoveStack.uEnd[ctx->uPly]; + x++) + { + mv = ctx->sMoveStack.mvf[x].mv; + if (TRUE == MakeMove(ctx, mv)) + { + UnmakeMove(ctx, mv); + return(TRUE); + } + } + return(FALSE); +} + +void +TestRecognExhaustiveKQKP(void) +/** + +Routine description: + + Exhaustively enumerate every legal KQKP (one queen, one pawn) + position -- not a random sample -- and verify _RecognizeKQKP's + verdict against direct EGTB probes wherever coverage exists. Same + "no counterexamples in N random samples is false confidence" + rationale as TestRecognExhaustiveKNKP/KPK -- doubly warranted here + since _RecognizeKQKP's win/UNRECOGNIZED boundary (rook/bishop pawn + on the square before queening) was scoped down from memory of + classical endgame theory rather than derived from EGTB data, and + this file has real precedent (_RecognizeKBKP's 2026-09-18 history) + for a plausible-looking classical shortcut turning out unsound once + fuzzed. + + Learned from TestRecognExhaustiveKPK's own 2026-09-18 history: + fixing which COLOR plays the strong side is NOT a safe assumption + in this codebase, even though RecognLookup/_RecognizeKQKP determine + "strong side" dynamically from piece counts -- iterates both + fBlackHasQueen assignments explicitly rather than trusting that + symmetry. Unlike TestRecognExhaustiveKPK, this keeps the left/right + mirror-halving restriction (FILE(cStrongKing) <= D, same as + TestRecognExhaustiveKNKP) -- safe here because, unlike _RecognizeKPK's + hand-typed critical-square table, _RecognizeKQKP contains no + per-square lookup data at all: its only file-dependent test is + "FILE(cPawn) == A/C/F/H", which treats both rook files and both + bishop files identically by construction, so a left/right board + reflection cannot expose an asymmetric transcription bug the way it + did for KPK's table. + + Not wired into TestRecogn()'s automatic startup suite (same + precommit_check.sh budget reasoning as KNKP/KPK) -- call this on + demand when validating a KQKP-area change. + +Parameters: + + void + +Return value: + + void + +**/ +{ + SEARCHER_THREAD_CONTEXT *ctx; + POSITION pos; + COOR cStrongKing, cWeakKing, cQueen, cPawn; + ULONG uToMove; + SCORE iScore, iEgtb; + ULONG uVal; + ULONG uTotal = 0, uChecked = 0, uSkipped = 0, uWrong = 0; + ULONG uReportedWrong = 0; + ULONG uWrongStalemated = 0; + FLAG fStalemated; + + Trace("Exhaustive KQKP (1 queen, 1 pawn) verification against " + "EGTB ground truth -- this takes a while...\n"); + ctx = malloc(sizeof(SEARCHER_THREAD_CONTEXT)); + if (NULL == ctx) + { + return; + } + InitializeSearcherContext(&pos, ctx); + + FOREACH_SQUARE(cStrongKing) + { + if (!IS_ON_BOARD(cStrongKing)) continue; + if (FILE(cStrongKing) > D) continue; + + FOREACH_SQUARE(cWeakKing) + { + if (!IS_ON_BOARD(cWeakKing)) continue; + if (DISTANCE(cStrongKing, cWeakKing) < 2) continue; + + FOREACH_SQUARE(cQueen) + { + if (!IS_ON_BOARD(cQueen)) continue; + if ((cQueen == cStrongKing) || (cQueen == cWeakKing)) continue; + + FOREACH_SQUARE(cPawn) + { + if (!IS_ON_BOARD(cPawn)) continue; + if (RANK1(cPawn) || RANK8(cPawn)) continue; + if ((cPawn == cStrongKing) || (cPawn == cWeakKing) || + (cPawn == cQueen)) continue; + + for (uToMove = BLACK; uToMove <= WHITE; uToMove++) + { + ULONG uColorPass; + + for (uColorPass = 0; uColorPass < 2; uColorPass++) + { + uTotal++; + if (0 == (uTotal % 500000)) + { + Trace(" ... %lu raw combos so far (%lu checked, " + "%lu wrong)\n", uTotal, uChecked, uWrong); + } + if (FALSE == _BuildKQKPPosition(&pos, cStrongKing, + cWeakKing, cQueen, + cPawn, uToMove, + (1 == uColorPass))) + { + continue; + } + ReInitializeSearcherContext(&pos, ctx); + uVal = _RecognizeKQKP(ctx, &iScore); + if (UNRECOGNIZED == uVal) + { + continue; + } + if (FALSE == ProbeEGTB(ctx, &iEgtb)) + { + uSkipped++; + continue; + } + uChecked++; + if (FALSE == _RecognizerAgreesWithEGTB(iScore, uVal, + iEgtb)) + { + uWrong++; + fStalemated = + (FALSE == InCheck(&ctx->sPosition, + ctx->sPosition.uToMove)) && + (FALSE == _ToMoveHasLegalMove(ctx)); + if (TRUE == fStalemated) + { + uWrongStalemated++; + } + if (uReportedWrong < 20) + { + Trace("MISMATCH (blackHasQueen=%lu) iScore=%d " + "uVal=%lu iEgtb=%d toMove=%lu " + "stalemated=%lu\n", + uColorPass, iScore, uVal, iEgtb, + pos.uToMove, fStalemated); + DumpPosition(&pos); + uReportedWrong++; + } + } + } + } + } + } + } + } + + { + ULONG uWrongOther = uWrong - uWrongStalemated; + + free(ctx); + Trace("Exhaustive KQKP: %lu raw combos, %lu checked against EGTB, " + "%lu skipped (no EGTB coverage), %lu WRONG (%lu of those are " + "the weak side simply stalemated, %lu are something else).\n", + uTotal, uChecked, uSkipped, uWrong, uWrongStalemated, + uWrongOther); + } +} + +// // Build a KPK (one pawn, one lone king) position directly from square // placements. fBlackHasPawn selects which color plays the pawn side // (cStrongKing/cWeakKing name roles, not colors, in both cases). @@ -845,7 +1106,7 @@ Return value: continue; } ReInitializeSearcherContext(&pos, ctx); - uVal = RecognLookup(ctx, &iScore, FALSE); + uVal = RecognLookup(ctx, &iScore, EGTB_PROBE_NEVER); if (UNRECOGNIZED == uVal) { continue; @@ -1227,7 +1488,7 @@ Return value: continue; } ReInitializeSearcherContext(&pos, ctx); - uVal = RecognLookup(ctx, &iScore, FALSE); + uVal = RecognLookup(ctx, &iScore, EGTB_PROBE_NEVER); if (UNRECOGNIZED == uVal) { uUnrecognized++; @@ -1758,7 +2019,7 @@ Return value: _MakeRandomEndgame(&pos, uPawnA, 0, 0, uPawnB, 0, 0); ReInitializeSearcherContext(&pos, ctx); - uVal = RecognLookup(ctx, &iScore, FALSE); + uVal = RecognLookup(ctx, &iScore, EGTB_PROBE_NEVER); if (UNRECOGNIZED == uVal) { uUnrecognized++; @@ -2011,7 +2272,7 @@ Return value: // signature routing a real search would use. // { - ULONG uVal = RecognLookup(ctx, &iRecScore, FALSE); + ULONG uVal = RecognLookup(ctx, &iRecScore, EGTB_PROBE_NEVER); if (UNRECOGNIZED != uVal) { uRecognClaimed++; |
