diff options
| author | Scott Gasch <[email protected]> | 2026-09-16 00:08:27 -0700 |
|---|---|---|
| committer | Scott Gasch <[email protected]> | 2026-09-16 00:08:27 -0700 |
| commit | b91c8453f3fe21c8ccd3568923d8e39141c9c945 (patch) | |
| tree | b908fea282ecd4075744590d3dfc88ffa903d943 /src | |
| parent | c0ac3d7845f60c7f6f2c3bfc306ceb0e4ac045ba (diff) | |
Add search self-tests (hash consistency, QSearch sanity) and a native EPD parser (`epd` command + ParseEpdRecord), fixing a silent-corruption bug in convert_epd.pl that broke every Black-to-move EPD conversion.
testsearch.c: TestSearchHashConsistency exercises the hash table through real search (repeat-with-warm-table and shallow-then-deep agreement) rather than synthetic store/probe calls. TestQSearchSanity checks termination and a node-count ceiling; its own header documents that the TEST=1 self-test build doesn't define PERF_COUNTERS, so it checks whole-search node count rather than the qsearch-only counter. A planned null-move zugzwang test was dropped in favor of tests/zugzwang.epd (best-move outcome, not an internal counter) after the three classic literature positions turned out not to structurally reach this engine's verification-search code path on this build.
fen.c/script.c/command.c: new `epd <file>` command reads raw EPD directly via ParseEpdRecord, reusing ScriptCommand's existing suite-reporting machinery (factored into _ProcessSuiteLine/_PrintSuiteReport). Replaces the need for tests/convert_epd.pl, which had a real bug: its bm/am regex's [^b]+ filler breaks on any Black-to-move record (side-to-move field is literally "b"), and unscoped $fen/$id/$solution globals meant a failed match silently duplicated the previous record instead of erroring.
tests/ep_to_epd.pl converts the existing setboard/solution/avoid/id/go suites back to real EPD; cross-checked ecm_ringers.ep_ via `script` against ecm_ringers.epd via `epd` -- byte-identical node counts and solve stats. Added .epd siblings for every genuine tactical suite in tests/.
Added src/TESTING_PLAN.md: full audit of test*.c coverage gaps (dead SEE oracle, unwired eval/recognizer tests, zero hash-table correctness coverage, thin search self-tests) with a prioritized plan for closing them.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Diffstat (limited to 'src')
| -rw-r--r-- | src/TESTING_PLAN.md | 253 | ||||
| -rwxr-xr-x | src/chess.h | 16 | ||||
| -rw-r--r-- | src/command.c | 6 | ||||
| -rw-r--r-- | src/fen.c | 216 | ||||
| -rw-r--r-- | src/main.c | 2 | ||||
| -rw-r--r-- | src/script.c | 378 | ||||
| -rw-r--r-- | src/testsearch.c | 298 |
7 files changed, 1053 insertions, 116 deletions
diff --git a/src/TESTING_PLAN.md b/src/TESTING_PLAN.md new file mode 100644 index 0000000..abf35e3 --- /dev/null +++ b/src/TESTING_PLAN.md @@ -0,0 +1,253 @@ +# Testing plan: closing coverage gaps in test*.c + +Written 2026-09-16 after an audit of every `src/test*.c` file and how +(or whether) each is wired into `main.c`'s self-test sequence. Goal +per Scott: not more test code for its own sake -- find the +years-old bugs that asserts have been missing. Priorities: search, +eval, generate (not exclusively). Perf testing is also explicitly +in scope. + +## Dead/disconnected infrastructure (fix first -- near-zero cost) + +Several tests already exist but don't run. Recovering this coverage is +strictly cheaper than writing anything new, so do this before anything +else below. + +1. **SEE has zero active correctness testing.** `testsee.c`'s own + header comment admits it: `DebugSEE`/`DebugSEERecursive` (a + brute-force make/unmake exchange-walk oracle meant to check `SEE()` + against) is wrapped in `#ifdef TEST_BROKEN` -- a define nothing + sets. `testsee.c`'s only *called* functions (`TestGetAttacks`, + `TestIsAttackedBB`) test attack generation, not SEE. Given SEE is + what `GetMoveSee`/extensions/QSearch pruning all trust (and is the + exact code reworked in commit c0ac3d7), this is the single + highest-value fix here. Get `TEST_BROKEN` -> live (`TEST`), fix + whatever bitrot surfaces, and confirm it's called from `main.c`. +2. **`TestEvalWithKnownPositions`** (`testeval.c`, ~20 hand-curated + "side X should be better here" positions from chess literature) is + defined but **never called from anywhere** -- not `main.c`, not + `command.c`. Wire it into the self-test sequence. +3. **`TestRecognExhaustiveKNKP`** (`testrecogn.c`, exhaustive KNKP + endgame-recognizer enumeration) is defined and prototyped in + `chess.h` but referenced only in a comment -- never invoked. Only + the smaller table-driven `TestRecogn` runs. Wire it in. +4. **The hash table has no correctness self-test at all.** + `testhash.c` contains `SanityCheckHashEntry` (an entry-shape + validator) and `AnalyzeFullHashTable` -- but the latter is a + post-search *diagnostic dump* called from `root.c`, not a test, and + it's never invoked from the `TEST` self-test sequence. See "New + test: hash correctness" below for what should replace this gap, not + just a wiring fix. +5. **`TestEvalWithSymmetry`** only runs a *single* random symmetric + position per invocation, and only when a human types `eval` at the + prompt in an `EVAL_DUMP` build (`command.c:2203`) -- not part of the + automated suite, never loops. See "New test: eval symmetry fuzz" + below. + +## New tests, by area + +### Search (thinnest area relative to its bug surface) + +- `TestSearch` (`testsearch.c`) runs 20 random positions at fixed + **depth 2** only, checking solely "returned move is legal" and + "didn't blow the time limit." + - **DONE -- hash-consistency test** (`TestSearchHashConsistency`, + testsearch.c): searches a position to depth D twice in a row with + a warm hash table (scores must match), and separately confirms a + depth-D2 search seeded by a shallower depth-D1 warm-up agrees with + a from-scratch depth-D2 search. Wired into the `TEST=1` self-test + sequence (main.c). + - **DONE (partially) -- QSearch sanity test** + (`TestQSearchSanity`, testsearch.c): checks search termination + (no `INVALID_SCORE`/garbage) and a whole-search node-count + ceiling. Caveat found and documented in the function's own header + comment: the self-test suite builds with plain `gmake TEST=1`, + which doesn't define `PERF_COUNTERS` -- every `sCounters` field + (including `u64QNodeCount`, the qsearch-only counter this was + originally meant to check) is wrapped in `chess.h`'s `INC()` macro, + a no-op without that define. Confirmed live (`sd 3; go` on a + `TEST`-only binary reports "0 qnodes" regardless of real qsearch + activity). The test currently checks + `ctx->sCounters.tree.u64TotalNodeCount` instead (root.c's + unconditional per-node `++`, always live) -- a real but strictly + weaker signal, since it can't distinguish "qsearch exploded" from + "full-width search exploded." Building a `PERF_COUNTERS`-enabled + `TEST` profile (or adding an always-live qsearch-only counter) is + the natural follow-up to close this for real. The hand-auditable- + tree idea (`Trace()` every qsearch node on a small position) is + still worth doing but wasn't implemented this session. + - **NOT DONE -- determinism test** (same position/depth searched + twice, PV must match): explicitly descoped -- Scott's call is that + a stable-PV requirement makes the test break every time move + ordering changes for unrelated reasons, which is just noise, not a + real regression signal. A same-*score* (not same-PV) check is + already covered by the hash-consistency test above. + - **NOT DONE -- MP-vs-single-thread agreement test**: still open. + `split.c` has no dedicated correctness test. + - **DONE (as an EPD suite, not a C test) -- null-move/zugzwang + soundness**: `tests/zugzwang.epd` (5 positions, run via the new + `epd` command -- see below) checks actual best-move outcome + instead of an internal counter. An earlier attempt asserted + against `ctx->sCounters.extension.uZugzwang` (the verification- + search hit counter) using the 3 classic literature zugzwang + positions, but none of them structurally reach that code path on + this build: 2 have too little material for + `WeShouldTryNullmovePruning`'s own `> 2` non-pawn-count gate to + attempt null-move pruning at all, and the 3rd (despite landing in + the eligible window) gets resolved directly off this box's local + Syzygy tablebase before search ever touches null-move logic -- + confirmed live via the `TABLEBASE MOVE` trace line. Removed in + favor of the EPD suite, which doesn't care about internal + mechanism, only the final answer. See `testsearch.c`'s comment + where `TestNullMoveZugzwang` used to be for the full writeup. + +**New infrastructure that came out of chasing the above (search.c +touches none of this, but it's now how any future EPD-based search +test should be added):** + +- **`epd <file>` engine command** (`EpdCommand`, script.c) parses and + runs a raw EPD suite file directly -- no perl preprocessing step + first. Backed by `ParseEpdRecord` (fen.c), real per-record EPD + parsing (4 position fields, optional FEN-style halfmove/fullmove + counters, then `;`-separated `bm`/`am`/`id` opcodes). +- **Found and fixed while building this**: `tests/convert_epd.pl` (the + old perl-based EPD-to-script converter) had a real, silent- + corruption bug -- its `bm`/`am` regex used `[^b]+` as the filler + between the board and the opcode, which breaks on *every* + Black-to-move record (the side-to-move field is literally the + character `b`), and since `$fen`/`$id`/`$solution` were unscoped + globals rather than per-line locals, a failed match silently carried + the *previous* record's values forward instead of erroring. + Confirmed live: feeding it a 5-position batch with one Black-to-move + entry produced a byte-for-byte duplicate of the previous record for + that one line, no diagnostic at all. The script is left in place + (marked superseded in its own header) in case something still shells + out to it, but new suites should use `epd` instead. + Cross-checked the new parser against the old pipeline on + `ecm_ringers`: `script` (old `.ep_` format) and `epd` (new + `tests/ecm_ringers.epd`, produced by `ep_to_epd.pl`, see next item) + produced byte-identical node counts and solve stats. +- **`tests/ep_to_epd.pl`**: one-off reverse converter, turns the + existing `setboard`/`solution`/`avoid`/`id`/`go` script-format + suites back into real EPD now that `epd` can read it directly. Run + against every genuine tactical suite in `tests/` (not `bench.ep_` or + `trial.ep_`, which use `sd`/`st`/`easy`/`quit` and aren't EPD + underneath) -- produced `.epd` siblings for `arasan`, + `ecm_confident_quick`, `ecm_confident_sample2`, `ecm_confident`, + `ecm_hard_quick`, `ecm_hard`, `ecm_late`, `ecm_quick`, `ecm_ringers`, + `ecm-gcp`, `ecm-hard`, `ecm`, `ecm016_only`, `LCT2`, `mes400`, + `newiq`, `quick`, `swiss`, `wac`. `ecm.epd` came out to 879 records, + matching the `606/879`/`622/879` solve-count baselines already + quoted elsewhere in CLAUDE.md. + +### Eval + +- Automate `TestEvalWithSymmetry` into a real fuzz loop (N random + symmetric positions, assert `Eval() == 0` for all, as part of the + standard self-test run) instead of the current single-shot + manual-only invocation. +- **Adversarial/degenerate-position term-bounds test**: 0-pawn + positions, 9-queen positions, fully-open-board positions -- extend + `GenerateRandomLegalPosition` or hand-build cases, assert no eval + term produces an out-of-range score (reuse the existing + `EvalTrace`/`EVAL_DUMP` term-tracing machinery for this). +- Audit for any **incrementally-maintained eval state** (pawn hash, + king-safety cache) and add a from-scratch-recompute-vs-incremental + consistency check if any exists -- classic "works until the hash + collides" bug class. +- **Eval speed regression test**: no benchmark currently exists for + eval itself despite it being the named NPS bottleneck vs. Crafty + (see CLAUDE.md). Same interleaved-comparison shape as the existing + `TestGenerate*Speed` benchmarks, broken down by game phase. + +### Generate (best-covered area, but gaps remain) + +- Perft (`TestMoveGenerator`) only checks to **ply 4** even though + `uLeaves[]` already has reference values to ply 6/7 sitting unused -- + extend to a `--perft-deep` slow path, and add perft-**divide** + (per-root-move leaf breakdown) to localize failures to a specific + move instead of just tripping a panic. +- Add perft positions specifically chosen to stress **en passant, + underpromotion, and castling-through-check** individually (the + current 5 FENs give general coverage but weren't chosen for this). +- `TestGenerateAllMovesSpeed` and the per-piece-type speed tests are + **pure microbenchmarks with no correctness assertion** -- they print + cycles and never diff the move *sets* produced by the mailbox vs. + bitboard generators being compared. Add a sorted-move-list-equality + assertion alongside each (both generators already run back-to-back + in the same loop, so this is nearly free) -- otherwise a bitboard + generator silently missing a move class just prints a suspiciously + fast number and passes. +- `GetMoveSee`'s new non-pawn-capture SEE path (c0ac3d7) has no + dedicated test yet -- covered by the SEE oracle fix above once live. + +### Move/Position plumbing + +- Reasonably solid already (`TestMakeUnmakeMove`, `TestExposesCheck`, + `TestIsAttacked`). Add a **long-game fuzz test**: play N random games + of ~200 plies each (not just 1-ply-at-a-time exercises), asserting + `VerifyPositionConsistency` and `ComputeSig`-from-scratch agreement + with the incrementally-maintained signature at every ply. Confirm + `VerifyPositionConsistency` is actually compiled live in the + `TEST`/`DEBUG` build, not just present in source. + +### Hash (currently zero correctness coverage) + +New test needed, not just a wiring fix: +- Store/probe roundtrip correctness. +- Replacement-scheme behavior under forced collisions (fill table past + capacity with distinct signatures, confirm replacement policy + behaves as designed, not just "doesn't crash"). +- `ADJUST_MATE_IN_N` ply-distance adjustment on `HASH_FLAG_EXACT` + entries specifically -- the exact mechanism CLAUDE.md flags as + audited-by-hand but not test-covered after this session's `ASSERT` + removals in `QSearch()`. + +### Recognizer + +- Wire in `TestRecognExhaustiveKNKP` (see dead-infra list above). + Consider the same exhaustive-enumeration treatment for any other + small recognized endgame class in `recogn.c` (KPK, KRK) if one + exists without a test. + +## Perf testing + +Current coverage is narrow: five isolated per-piece-type +mailbox-vs-bitboard microbenchmarks (`TestGenerate*Speed`), nothing for +search, eval, or hash. + +- **Eval NPS/cycles-per-call benchmark**, same interleaved-comparison + shape as the existing generate benchmarks, by game phase -- eval is + the named bottleneck (CLAUDE.md's 1.5M vs 7.5M nps gap vs. Crafty) + and has no dedicated speed test today. +- **Whole-search NPS benchmark**: `bench.c` already exists for this -- + confirm whether it's part of the automated `TEST` self-test pass or + purely a manual `bench` command; if manual-only, wire a + fixed-node-budget version into the automated sequence so a + regression surfaces without someone remembering to run it by hand. +- **Hash-probe-under-load benchmark**: cycles/probe at low occupancy + vs. near-full table -- replacement-scheme cost can regress silently + today with nothing to measure it. +- **Stored baselines with pass/fail thresholds**, not print-only + output: today every perf test prints cycles/call and leaves + interpretation to a human reading the log. Store a last-known-good + baseline (same idea as `head_reference/` for solve-counts) and fail + loud if a call site regresses past some tolerance. + +## Suggested priority order + +1. Un-bitrot the three dead tests (SEE oracle, + `TestEvalWithKnownPositions`, `TestRecognExhaustiveKNKP`) -- + near-zero cost, immediate coverage recovery. +2. Hash-table correctness test (store/probe/replace/mate-adjust) -- + currently the single biggest correctness blind spot. +3. Search determinism + MP-vs-single-thread agreement test -- targets + the exact class of bug most likely to hide in threading/hash/ + extension interactions that per-function unit tests can't see. +4. Extend perft to deeper plies + divide output; add move-set-equality + assertions to the existing generate speed benchmarks (turns + dead-weight microbenchmarks into correctness tests for free). +5. Eval: automate the symmetry fuzz test (loop N, not 1) and add + adversarial/degenerate-position term-bounds checks. +6. Round out perf testing per the list above, with stored baselines + and pass/fail thresholds instead of print-only output. diff --git a/src/chess.h b/src/chess.h index 16cdca9..38fdb3f 100755 --- a/src/chess.h +++ b/src/chess.h @@ -2162,6 +2162,15 @@ PositionToFen(POSITION *p); FLAG FenToPosition(POSITION *p, char *szFen); +FLAG +ParseEpdRecord(char *szLineIn, + char *szFen, ULONG cbFen, + char *szId, ULONG cbId, + char szMoves[][SMALL_STRING_LEN_CHAR], + ULONG uMaxMoves, + ULONG *puNumMoves, + FLAG *pfIsAvoid); + // // testfen.c // @@ -3051,6 +3060,7 @@ IsLegalDrawByRepetition(void); // script.c // COMMAND(ScriptCommand); +COMMAND(EpdCommand); COMMAND(SolutionCommand); @@ -3303,6 +3313,12 @@ SanityCheckMoves(IN SEARCHER_THREAD_CONTEXT *ctx, FLAG TestSearch(void); +FLAG +TestSearchHashConsistency(void); + +FLAG +TestQSearchSanity(void); + // // see.c // diff --git a/src/command.c b/src/command.c index 0662333..5087aee 100644 --- a/src/command.c +++ b/src/command.c @@ -2720,6 +2720,12 @@ COMMAND_PARSER_ENTRY g_ParserTable[] = FALSE, TRUE, "Inform the engine about a game result" }, + { "epd", + EpdCommand, + FALSE, + FALSE, + FALSE, + "Run a raw EPD test suite file directly (no conversion needed)" }, { "script", ScriptCommand, FALSE, @@ -781,3 +781,219 @@ Return value: SystemFreeMemory(szCapturedFen); return(TRUE); } + + +FLAG +ParseEpdRecord(IN char *szLineIn, + OUT char *szFen, IN ULONG cbFen, + OUT char *szId, IN ULONG cbId, + OUT char szMoves[][SMALL_STRING_LEN_CHAR], + IN ULONG uMaxMoves, + OUT ULONG *puNumMoves, + OUT FLAG *pfIsAvoid) +/** + +Routine description: + + Real per-record EPD parsing: pull the board/side-to-move/castling/ + en-passant fields plus the "bm"/"am" and "id" opcodes out of one + raw EPD line, e.g.: + + 8/8/1p1r1k2/p1pPN1p1/P3KnP1/1P6/8/3R4 b - - 0 1 bm Nxd5; id "zugzwang.005"; + + Written to replace tests/convert_epd.pl's single-regex approach + ("everything between the board and bm is [^b]+"), which silently + breaks on exactly this kind of line: side-to-move "b" contains the + character the regex's filler class explicitly excludes, so the + match fails outright, and because that script's $fen/$id/$solution + were plain globals rather than per-line locals, a failed match + silently carried the *previous* line's values forward instead of + erroring. This function fails closed instead: a line it can't + parse returns FALSE and touches none of the OUT parameters, so a + caller looping over a file can never mistake stale data left over + from the previous line for a real result. + + Tolerates the two EPD-ish flavors seen in real suites: strict EPD + (exactly 4 position fields, then opcodes) and the more common + FEN-plus-opcodes hybrid (4 position fields, then halfmove/fullmove + counters, then opcodes) -- an opcode name is never purely numeric, + so the two are unambiguous to tell apart by just skipping leading + all-digit tokens before looking for "bm"/"am"/"id". + +Parameters: + + IN char *szLineIn : one raw EPD line (not modified) + OUT char *szFen : receives "<board> <stm> <castle> <ep>" + IN ULONG cbFen : size of szFen + OUT char *szId : receives the id string (unquoted), or "" if none + IN ULONG cbId : size of szId + OUT char szMoves[][] : receives each bm/am move token, in order + IN ULONG uMaxMoves : capacity of szMoves (row count) + OUT ULONG *puNumMoves : number of moves written to szMoves + OUT FLAG *pfIsAvoid : TRUE if the opcode was "am", FALSE if "bm" + +Return value: + + FLAG : TRUE if a fen and at least one bm/am move were found and + fit in the caller's buffers; FALSE otherwise (all OUT + parameters left untouched) + +**/ +{ + char szWork[SMALL_STRING_LEN_CHAR]; + char *szTokens[4]; + char *p; + char *szOpcodeSection; + char *szChunk; + char *szSavePtr1, *szSavePtr2; + ULONG u; + ULONG uNumMoves = 0; + FLAG fIsAvoid = FALSE; + char szLocalId[SMALL_STRING_LEN_CHAR]; + char szLocalFen[SMALL_STRING_LEN_CHAR]; + + szLocalId[0] = '\0'; + + if ((NULL == szLineIn) || ('\0' == szLineIn[0])) + { + return(FALSE); + } + strncpy(szWork, szLineIn, ARRAY_LENGTH(szWork) - 1); + szWork[ARRAY_LENGTH(szWork) - 1] = '\0'; + szWork[strcspn(szWork, "\r\n")] = '\0'; + + p = szWork; + while (isspace((unsigned char)*p)) p++; + if (('\0' == *p) || ('#' == *p)) + { + return(FALSE); + } + + // First 4 whitespace-delimited tokens are always the position. + for (u = 0; u < 4; u++) + { + while (isspace((unsigned char)*p)) p++; + if ('\0' == *p) + { + return(FALSE); + } + szTokens[u] = p; + while (*p && !isspace((unsigned char)*p)) p++; + if (*p) + { + *p = '\0'; + p++; + } + } + if (snprintf(szLocalFen, ARRAY_LENGTH(szLocalFen), "%s %s %s %s", + szTokens[0], szTokens[1], szTokens[2], szTokens[3]) < 0) + { + return(FALSE); + } + + // Skip FEN-style halfmove/fullmove counters ahead of the first + // real opcode -- an opcode name is never purely numeric. + for (;;) + { + char *q; + while (isspace((unsigned char)*p)) p++; + q = p; + if ('\0' == *q) + { + break; + } + while (isdigit((unsigned char)*q)) q++; + if ((q == p) || (*q && !isspace((unsigned char)*q) && (';' != *q))) + { + // Not a bare numeric token -- this is the opcode section. + break; + } + if ('\0' == *q) + { + p = q; + break; + } + p = q; + } + szOpcodeSection = p; + + // Now split what's left on ';' into individual opcode chunks. + for (szChunk = strtok_r(szOpcodeSection, ";", &szSavePtr1); + NULL != szChunk; + szChunk = strtok_r(NULL, ";", &szSavePtr1)) + { + char *szTrimmed = szChunk; + char *szTail; + + while (isspace((unsigned char)*szTrimmed)) szTrimmed++; + szTail = szTrimmed + strlen(szTrimmed); + while ((szTail > szTrimmed) && isspace((unsigned char)szTail[-1])) + { + szTail--; + *szTail = '\0'; + } + if ('\0' == *szTrimmed) + { + continue; + } + + if (((0 == STRNCMPI(szTrimmed, "bm", 2)) && + isspace((unsigned char)szTrimmed[2])) || + ((0 == STRNCMPI(szTrimmed, "am", 2)) && + isspace((unsigned char)szTrimmed[2]))) + { + char *szMoveTok; + + fIsAvoid = (0 == STRNCMPI(szTrimmed, "am", 2)); + szMoveTok = strtok_r(szTrimmed + 2, " \t", &szSavePtr2); + while (NULL != szMoveTok) + { + if (uNumMoves < uMaxMoves) + { + strncpy(szMoves[uNumMoves], szMoveTok, + SMALL_STRING_LEN_CHAR - 1); + szMoves[uNumMoves][SMALL_STRING_LEN_CHAR - 1] = '\0'; + uNumMoves++; + } + szMoveTok = strtok_r(NULL, " \t", &szSavePtr2); + } + } + else if ((0 == STRNCMPI(szTrimmed, "id", 2)) && + isspace((unsigned char)szTrimmed[2])) + { + char *szIdVal = szTrimmed + 2; + while (isspace((unsigned char)*szIdVal)) szIdVal++; + if ('"' == *szIdVal) + { + char *szEndQuote; + szIdVal++; + szEndQuote = strchr(szIdVal, '"'); + if (NULL != szEndQuote) + { + *szEndQuote = '\0'; + } + } + strncpy(szLocalId, szIdVal, ARRAY_LENGTH(szLocalId) - 1); + szLocalId[ARRAY_LENGTH(szLocalId) - 1] = '\0'; + } + // Other opcodes (acd, ce, dm, c0, ...) intentionally ignored. + } + + if (0 == uNumMoves) + { + return(FALSE); + } + if (strlen(szLocalFen) >= cbFen) + { + return(FALSE); + } + strcpy(szFen, szLocalFen); + if (strlen(szLocalId) >= cbId) + { + szLocalId[cbId - 1] = '\0'; + } + strcpy(szId, szLocalId); + *puNumMoves = uNumMoves; + *pfIsAvoid = fIsAvoid; + return(TRUE); +} @@ -619,6 +619,8 @@ Return value: TestIsAttacked(); TestMakeUnmakeMove(); TestSearch(); + TestSearchHashConsistency(); + TestQSearchSanity(); return(TRUE); } diff --git a/src/script.c b/src/script.c index 0bc09e4..e8d6f9a 100644 --- a/src/script.c +++ b/src/script.c @@ -349,6 +349,160 @@ Return value: +// +// Shared by ScriptCommand and EpdCommand: process one already-read +// line exactly the way a script file's line is processed -- "go" +// triggers a search plus counter accumulation, anything else is +// dispatched through the normal command parser. Factored out so +// EpdCommand can feed it lines synthesized from a raw EPD record +// (setboard/id/solution/avoid/go) instead of lines read verbatim from +// a file, without duplicating this logic or its counter bookkeeping. +// +static void +_ProcessSuiteLine(char *szLine, POSITION *pos) +{ + Trace("SCRIPT> %s\n", szLine); + if (!STRNCMPI(szLine, "go", 2)) + { + (void)Think(pos); + g_SuiteCounters.u64TotalNodeCount += + g_Options.u64NodesSearched; + g_SuiteCounters.u64TotalBetaCutoffs += + g_Options.u64BetaCutoffs; + g_SuiteCounters.u64TotalBetaCutoffsOnFirstMove += + g_Options.u64BetaCutoffsOnFirstMove; + g_SuiteCounters.u64TotalCounterMoveTries += + g_Options.u64CounterMoveTries; + g_SuiteCounters.u64TotalCounterMoveHits += + g_Options.u64CounterMoveHits; + g_SuiteCounters.u64TotalLeftoverTries += + g_Options.u64LeftoverTries; + g_SuiteCounters.u64TotalLeftoverAlpha += + g_Options.u64LeftoverAlpha; + g_SuiteCounters.u64TotalLeftoverFH += + g_Options.u64LeftoverFH; + { + ULONG v; + for (v = 0; v < CM_EVIDENCE_BUCKETS; v++) + { + g_SuiteCounters.u64TotalCMEvidenceTries[v] += + g_Options.u64CMEvidenceTries[v]; + g_SuiteCounters.u64TotalCMEvidenceFH[v] += + g_Options.u64CMEvidenceFH[v]; + } + } + } + else + { + PushNewInput(szLine); + if (FALSE == g_fExitProgram) + { + ParseUserInput(FALSE); + } + } +} + + +// +// Shared by ScriptCommand and EpdCommand: the end-of-suite banner and +// statistics report. Verbatim extraction of what used to be +// ScriptCommand's own tail -- see that function's history for the +// meaning of each field. +// +static void +_PrintSuiteReport(double dSuiteStart) +{ + ULONG u, v, uMax; + double dMult = (double)g_Options.uMyIncrement / (double)SUITE_NUM_HISTOGRAM; + + Banner(); + Trace("\n" + "TEST SCRIPT execution complete. Final statistics:\n" + "--------------------------------------------------\n" + " correct solutions : %u\n" + " incorrect solutions : %u\n" + " total problems : %u\n" + " total nodecount : %" + COMPILER_LONGLONG_UNSIGNED_FORMAT "\n" + " avg. search speed : %6.1f nps\n" + " avg. solution time : %3.1f sec\n" + " avg. search depth : %4.1f ply\n" + " 1st move beta cut : %5.2f percent\n" + " avg. eff. branching : %5.3f\n" + " counter move hit %% : %5.2f percent (%" + COMPILER_LONGLONG_UNSIGNED_FORMAT " tries)\n" + " leftover : %5.2f%% alpha, %5.2f%% FH (%" + COMPILER_LONGLONG_UNSIGNED_FORMAT " tries)\n" + " script time : %6.1f sec\n\n", + g_SuiteCounters.uCorrect, + g_SuiteCounters.uIncorrect, + g_SuiteCounters.uTotal, + g_SuiteCounters.u64TotalNodeCount, + ((double)g_SuiteCounters.u64TotalNodeCount / + (SystemTimeStamp() - dSuiteStart)), + ((g_SuiteCounters.uCorrect > 0) ? + (g_SuiteCounters.dSigmaSolutionTime / + (double)g_SuiteCounters.uCorrect) : 0.0), + ((double)g_SuiteCounters.uSigmaDepth / + (double)g_SuiteCounters.uTotal), + (100.0 * (double)g_SuiteCounters.u64TotalBetaCutoffsOnFirstMove / + ((double)g_SuiteCounters.u64TotalBetaCutoffs + 1.0)), + (g_SuiteCounters.dSigmaEBF / + ((double)g_SuiteCounters.uEBFCount + 1.0)), + (100.0 * (double)g_SuiteCounters.u64TotalCounterMoveHits / + ((double)g_SuiteCounters.u64TotalCounterMoveTries + 1.0)), + g_SuiteCounters.u64TotalCounterMoveTries, + (100.0 * (double)g_SuiteCounters.u64TotalLeftoverAlpha / + ((double)g_SuiteCounters.u64TotalLeftoverTries + 1.0)), + (100.0 * (double)g_SuiteCounters.u64TotalLeftoverFH / + ((double)g_SuiteCounters.u64TotalLeftoverTries + 1.0)), + g_SuiteCounters.u64TotalLeftoverTries, + (SystemTimeStamp() - dSuiteStart)); + + // Countermove evidence calibration -- FH% per evidence bucket, + // an ongoing check that COUNTERMOVE_EVIDENCE_THRESHOLD (chess.h) + // is still well-calibrated as the engine/data evolve. + { + static const ULONG uCMEvidenceFloors[CM_EVIDENCE_BUCKETS] = + { 0, 1, 100, 1000, 10000, 100000, 1000000 }; + Trace("\ncountermove evidence calibration:\n"); + for (v = 0; v < CM_EVIDENCE_BUCKETS; v++) + { + Trace(" evidence >= %8lu : %5.2f%% FH (%" + COMPILER_LONGLONG_UNSIGNED_FORMAT " tries)\n", + uCMEvidenceFloors[v], + (100.0 * (double)g_SuiteCounters.u64TotalCMEvidenceFH[v] / + ((double)g_SuiteCounters.u64TotalCMEvidenceTries[v] + 1.0)), + g_SuiteCounters.u64TotalCMEvidenceTries[v]); + } + } + + // Histogram stuff + if (g_SuiteCounters.uTotal > 0) { + uMax = g_SuiteCounters.uHistogram[0]; + for (u = 1; u < SUITE_NUM_HISTOGRAM; u++) + { + if (g_SuiteCounters.uHistogram[u] > uMax) + { + uMax = g_SuiteCounters.uHistogram[u]; + } + } + for (u = 0; u < SUITE_NUM_HISTOGRAM; u++) + { + Trace("%4.1f .. %4.1f: ", + (double)u * dMult, (double)(u + 1) * dMult); + for (v = 0; + (uMax > 0) && (v < 50 * g_SuiteCounters.uHistogram[u] / uMax); + v++) + { + Trace("*"); + } + Trace("\n"); + } + } +} + + COMMAND(ScriptCommand) /** @@ -389,8 +543,6 @@ Return value: FILE *p; static CHAR szLine[SMALL_STRING_LEN_CHAR]; double dSuiteStart; - ULONG u, v, uMax; - double dMult; if (argc < 2) { @@ -402,143 +554,137 @@ Return value: Trace("Error (file doesn't exist): %s\n", argv[1]); return; } - + p = fopen(argv[1], "rb"); if (NULL != p) { - // Doing the script... dSuiteStart = SystemTimeStamp(); memset(&(g_SuiteCounters), 0, sizeof(g_SuiteCounters)); while(fgets(szLine, ARRAY_LENGTH(szLine), p)) { - Trace("SCRIPT> %s\n", szLine); - if (!STRNCMPI(szLine, "go", 2)) - { - (void)Think(pos); - g_SuiteCounters.u64TotalNodeCount += - g_Options.u64NodesSearched; - g_SuiteCounters.u64TotalBetaCutoffs += - g_Options.u64BetaCutoffs; - g_SuiteCounters.u64TotalBetaCutoffsOnFirstMove += - g_Options.u64BetaCutoffsOnFirstMove; - g_SuiteCounters.u64TotalCounterMoveTries += - g_Options.u64CounterMoveTries; - g_SuiteCounters.u64TotalCounterMoveHits += - g_Options.u64CounterMoveHits; - g_SuiteCounters.u64TotalLeftoverTries += - g_Options.u64LeftoverTries; - g_SuiteCounters.u64TotalLeftoverAlpha += - g_Options.u64LeftoverAlpha; - g_SuiteCounters.u64TotalLeftoverFH += - g_Options.u64LeftoverFH; - for (v = 0; v < CM_EVIDENCE_BUCKETS; v++) - { - g_SuiteCounters.u64TotalCMEvidenceTries[v] += - g_Options.u64CMEvidenceTries[v]; - g_SuiteCounters.u64TotalCMEvidenceFH[v] += - g_Options.u64CMEvidenceFH[v]; - } - } - else - { - PushNewInput(szLine); - if (TRUE == g_fExitProgram) break; - ParseUserInput(FALSE); - } + _ProcessSuiteLine(szLine, pos); + if (TRUE == g_fExitProgram) break; } fclose(p); + _PrintSuiteReport(dSuiteStart); + } +} - // Banner and end results - dMult = (double)g_Options.uMyIncrement / (double)SUITE_NUM_HISTOGRAM; - Banner(); - Trace("\n" - "TEST SCRIPT execution complete. Final statistics:\n" - "--------------------------------------------------\n" - " correct solutions : %u\n" - " incorrect solutions : %u\n" - " total problems : %u\n" - " total nodecount : %" - COMPILER_LONGLONG_UNSIGNED_FORMAT "\n" - " avg. search speed : %6.1f nps\n" - " avg. solution time : %3.1f sec\n" - " avg. search depth : %4.1f ply\n" - " 1st move beta cut : %5.2f percent\n" - " avg. eff. branching : %5.3f\n" - " counter move hit %% : %5.2f percent (%" - COMPILER_LONGLONG_UNSIGNED_FORMAT " tries)\n" - " leftover : %5.2f%% alpha, %5.2f%% FH (%" - COMPILER_LONGLONG_UNSIGNED_FORMAT " tries)\n" - " script time : %6.1f sec\n\n", - g_SuiteCounters.uCorrect, - g_SuiteCounters.uIncorrect, - g_SuiteCounters.uTotal, - g_SuiteCounters.u64TotalNodeCount, - ((double)g_SuiteCounters.u64TotalNodeCount / - (SystemTimeStamp() - dSuiteStart)), - ((g_SuiteCounters.uCorrect > 0) ? - (g_SuiteCounters.dSigmaSolutionTime / - (double)g_SuiteCounters.uCorrect) : 0.0), - ((double)g_SuiteCounters.uSigmaDepth / - (double)g_SuiteCounters.uTotal), - (100.0 * (double)g_SuiteCounters.u64TotalBetaCutoffsOnFirstMove / - ((double)g_SuiteCounters.u64TotalBetaCutoffs + 1.0)), - (g_SuiteCounters.dSigmaEBF / - ((double)g_SuiteCounters.uEBFCount + 1.0)), - (100.0 * (double)g_SuiteCounters.u64TotalCounterMoveHits / - ((double)g_SuiteCounters.u64TotalCounterMoveTries + 1.0)), - g_SuiteCounters.u64TotalCounterMoveTries, - (100.0 * (double)g_SuiteCounters.u64TotalLeftoverAlpha / - ((double)g_SuiteCounters.u64TotalLeftoverTries + 1.0)), - (100.0 * (double)g_SuiteCounters.u64TotalLeftoverFH / - ((double)g_SuiteCounters.u64TotalLeftoverTries + 1.0)), - g_SuiteCounters.u64TotalLeftoverTries, - (SystemTimeStamp() - dSuiteStart)); - // Countermove evidence calibration -- FH% per evidence bucket, - // an ongoing check that COUNTERMOVE_EVIDENCE_THRESHOLD (chess.h) - // is still well-calibrated as the engine/data evolve. +COMMAND(EpdCommand) +/** + +Routine description: + + Runs a raw EPD test suite file directly -- no perl preprocessing + into a setboard/solution/go script first. Historically the only + way to run a suite from real-world EPD (the "board stm castle ep + [halfmove fullmove] bm <move>; id \"...\"; ..." format most public + tactical suites ship in) was tests/convert_epd.pl, which had a + real, silent-corruption bug: its single "everything between the + board and bm is [^b]+" regex breaks on any Black-to-move record + (the side-to-move field is literally the character "b"), and + because $fen/$id/$solution were unscoped globals rather than + per-line locals, a failed match silently carried the *previous* + record's board and solution forward instead of erroring -- + confirmed live by feeding it a real 5-position batch, where the + one Black-to-move line came out as a byte-for-byte duplicate of + the line before it. This command replaces that whole external + preprocessing step with real per-record EPD parsing (see + ParseEpdRecord, fen.c) feeding the exact same + _ProcessSuiteLine/_PrintSuiteReport machinery ScriptCommand uses, + so a raw EPD file can just be handed to the engine directly. + + Usage: + + epd <required filename> + +Parameters: + + The COMMAND macro hides four arguments from the input parser: + + CHAR *szInput : the full line of input + ULONG argc : number of argument chunks + CHAR *argv[] : array of ptrs to each argument chunk + POSITION *pos : a POSITION pointer to operate on + +Return value: + + void + +**/ +{ + FILE *p; + static CHAR szLine[SMALL_STRING_LEN_CHAR]; + static CHAR szSynth[SMALL_STRING_LEN_CHAR]; + CHAR szFen[SMALL_STRING_LEN_CHAR]; + CHAR szId[SMALL_STRING_LEN_CHAR]; + CHAR szMoves[SUITE_MAX_SOLUTIONS + SUITE_MAX_AVOIDS][SMALL_STRING_LEN_CHAR]; + ULONG uNumMoves; + FLAG fIsAvoid; + double dSuiteStart; + ULONG u; + ULONG uLineNum = 0; + + if (argc < 2) + { + Trace("Error (Usage: epd <filename>): %s\n", argv[0]); + return; + } + if (FALSE == SystemDoesFileExist(argv[1])) + { + Trace("Error (file doesn't exist): %s\n", argv[1]); + return; + } + + p = fopen(argv[1], "rb"); + if (NULL != p) + { + dSuiteStart = SystemTimeStamp(); + memset(&(g_SuiteCounters), 0, sizeof(g_SuiteCounters)); + while(fgets(szLine, ARRAY_LENGTH(szLine), p)) { - static const ULONG uCMEvidenceFloors[CM_EVIDENCE_BUCKETS] = - { 0, 1, 100, 1000, 10000, 100000, 1000000 }; - Trace("\ncountermove evidence calibration:\n"); - for (v = 0; v < CM_EVIDENCE_BUCKETS; v++) + uLineNum++; + if (FALSE == ParseEpdRecord(szLine, szFen, ARRAY_LENGTH(szFen), + szId, ARRAY_LENGTH(szId), + szMoves, ARRAY_LENGTH(szMoves), + &uNumMoves, &fIsAvoid)) { - Trace(" evidence >= %8lu : %5.2f%% FH (%" - COMPILER_LONGLONG_UNSIGNED_FORMAT " tries)\n", - uCMEvidenceFloors[v], - (100.0 * (double)g_SuiteCounters.u64TotalCMEvidenceFH[v] / - ((double)g_SuiteCounters.u64TotalCMEvidenceTries[v] + 1.0)), - g_SuiteCounters.u64TotalCMEvidenceTries[v]); + if (szLine[0] && (szLine[0] != '\n') && (szLine[0] != '#')) + { + Trace("epd: skipping unparseable line %lu: %s", + uLineNum, szLine); + } + continue; } - } - // Histogram stuff - if (g_SuiteCounters.uTotal > 0) { - uMax = g_SuiteCounters.uHistogram[0]; - for (u = 1; u < SUITE_NUM_HISTOGRAM; u++) + snprintf(szSynth, ARRAY_LENGTH(szSynth), "setboard %s\n", szFen); + _ProcessSuiteLine(szSynth, pos); + if (TRUE == g_fExitProgram) break; + + if (szId[0]) { - if (g_SuiteCounters.uHistogram[u] > uMax) - { - uMax = g_SuiteCounters.uHistogram[u]; - } + snprintf(szSynth, ARRAY_LENGTH(szSynth), "id %s\n", szId); + _ProcessSuiteLine(szSynth, pos); } - for (u = 0; u < SUITE_NUM_HISTOGRAM; u++) + + for (u = 0; u < uNumMoves; u++) { - Trace("%4.1f .. %4.1f: ", - (double)u * dMult, (double)(u + 1) * dMult); - for (v = 0; - (uMax > 0) && (v < 50 * g_SuiteCounters.uHistogram[u] / uMax); - v++) - { - Trace("*"); - } - Trace("\n"); + snprintf(szSynth, ARRAY_LENGTH(szSynth), "%s %s\n", + fIsAvoid ? "avoid" : "solution", szMoves[u]); + _ProcessSuiteLine(szSynth, pos); } + + _ProcessSuiteLine("go\n", pos); + if (TRUE == g_fExitProgram) break; } + fclose(p); + _PrintSuiteReport(dSuiteStart); } } -FLAG + +FLAG WeAreRunningASuite(void) /** diff --git a/src/testsearch.c b/src/testsearch.c index 3d560ac..6b43f64 100644 --- a/src/testsearch.c +++ b/src/testsearch.c @@ -131,4 +131,302 @@ TestSearch(void) SystemFreeMemory(ctx); return(fRet); } + + +// +// Shared setup for the hash-consistency and zugzwang tests below: run +// one fixed-depth search on a fresh SEARCHER_THREAD_CONTEXT-relative +// position (ctx is caller-owned and reused across calls, matching +// TestSearch's own pattern -- SystemAllocateMemory'ing a fresh 75MB +// context per call would be wasteful for tests that need many runs +// over the same handful of positions) and return the root score. +// Does NOT touch the hash table itself (no Clear/Dirty call) -- that +// decision is the caller's, since the whole point of the hash tests +// below is controlling exactly what's in the table before each call. +// +static SCORE +_FixedDepthSearchScore(SEARCHER_THREAD_CONTEXT *ctx, + POSITION *pos, + ULONG uMaxDepth) +{ + GAME_RESULT result; + FLAG fPost = g_Options.fShouldPost; + + g_Options.fShouldPost = FALSE; + InitializeSearcherContext(pos, ctx); + g_MoveTimer.bvFlags = 0; + g_Options.fPondering = FALSE; + g_Options.fThinking = TRUE; + g_Options.fSuccessfulPonder = FALSE; + MaintainDynamicMoveOrdering(); + SetMoveTimerForTestingSearch(); + g_Options.uMaxDepth = uMaxDepth; +#if (PERF_COUNTERS && MP) + ClearHelperThreadIdleness(); +#endif + result = Iterate(ctx); + g_Options.fShouldPost = fPost; + if (RESULT_IN_PROGRESS != result.eResult) + { + // Mate/stalemate/draw at the root -- no root score was set by + // this iteration (there's no move to score). Not a bug, just + // not a comparable data point for the callers below. + return(INVALID_SCORE); + } + return(ctx->iRootScore); +} + + +// +// Hash-table correctness, exercised through real search rather than +// synthetic store/probe calls -- the actual bug class this is meant +// to catch (stale entries surviving a generation boundary, a bound +// getting misapplied, ADJUST_MATE_IN_N misfiring on an entry that +// shouldn't get it) only shows up when a search's outcome actually +// depends on what it reads back out of the table, which a +// hand-constructed store()-then-probe() unit test can't exercise -- +// see TESTING_PLAN.md's "hash correctness" section for the case for a +// synthetic store/probe test too; this is the complementary +// behavioral half, not a replacement for it. +// +// Two invariants, neither of which cares about search internals +// (move ordering, PV, node count) changing across future search work +// -- only that the table is *transparent*, i.e. its presence changes +// performance, never the answer: +// +// 1. Searching the same position to the same depth twice in a row +// (second time with the table already populated by the first) +// must produce the same score both times. +// 2. Searching to depth D2 with the table pre-populated by a +// shallower depth-D1 search of the same position must produce +// the same score as searching directly to D2 from a freshly +// cleared table. A depth-D1 entry read back during the depth-D2 +// search is only ever a bound, never trusted as exact at a +// higher depth than it was stored at -- if it were, this would +// catch it. +// +FLAG +TestSearchHashConsistency(void) +{ + static const char *rgszFen[] = + { + "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1", + "r1bq1rk1/pp2bppp/2n1pn2/2pp4/3P4/2NBPN2/PP3PPP/R1BQ1RK1 w - - 0 1", + "r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 1", + "8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 1", + }; + const ULONG uD1 = 4; + const ULONG uD2 = 6; + SEARCHER_THREAD_CONTEXT *ctx; + POSITION pos; + SCORE iFirst, iSecond, iShallow, iDirect; + ULONG u; + FLAG fRet = TRUE; + + Trace("Testing hash table consistency through search...\n"); + ctx = SystemAllocateMemory(sizeof(SEARCHER_THREAD_CONTEXT)); + ASSERT(ctx); + + for (u = 0; u < ARRAY_LENGTH(rgszFen); u++) + { + if (FALSE == FenToPosition(&pos, (char *)rgszFen[u])) + { + UtilPanic(INCONSISTENT_STATE, + NULL, NULL, NULL, NULL, + __FILE__, __LINE__); + } + + // Invariant 1: repeat-with-warm-table stability. + ClearHashTable(); + DirtyHashTable(); + iFirst = _FixedDepthSearchScore(ctx, &pos, uD1); + DirtyHashTable(); + iSecond = _FixedDepthSearchScore(ctx, &pos, uD1); + if ((INVALID_SCORE != iFirst) && + (INVALID_SCORE != iSecond) && + (iFirst != iSecond)) + { + Trace("TestSearchHashConsistency: position %lu: same-depth " + "rerun with warm hash table diverged (%d vs %d) -- " + "FEN: %s\n", u, iFirst, iSecond, rgszFen[u]); + fRet = FALSE; + } + + // Invariant 2: a shallower pre-population must not corrupt a + // deeper search's answer. + ClearHashTable(); + DirtyHashTable(); + iShallow = _FixedDepthSearchScore(ctx, &pos, uD1); + DirtyHashTable(); + iSecond = _FixedDepthSearchScore(ctx, &pos, uD2); + + ClearHashTable(); + DirtyHashTable(); + iDirect = _FixedDepthSearchScore(ctx, &pos, uD2); + + if ((INVALID_SCORE != iSecond) && + (INVALID_SCORE != iDirect) && + (iSecond != iDirect)) + { + Trace("TestSearchHashConsistency: position %lu: depth-%lu " + "search seeded by a depth-%lu warm-up (%d) disagreed " + "with a from-scratch depth-%lu search (%d) -- FEN: " + "%s\n", u, uD2, uD1, iSecond, uD2, iDirect, + rgszFen[u]); + fRet = FALSE; + } + (void)iShallow; + } + + if (FALSE == fRet) + { + UtilPanic(TESTCASE_FAILURE, + NULL, "TestSearchHashConsistency", NULL, NULL, + __FILE__, __LINE__); + } + ClearHashTable(); + SystemFreeMemory(ctx); + return(fRet); +} + + +// Null-move soundness is intentionally NOT tested here as a C unit +// test. An earlier version of this file asserted against +// ctx->sCounters.extension.uZugzwang (searchsup.c's verification- +// search counter) using the three classic "verified null-move +// pruning" zugzwang positions -- but on this codebase's own material +// gates (WeShouldTryNullmovePruning's `> 2` non-pawn-count guard, +// TryNullmovePruning's `< 4` verification window) and this box's +// Syzygy coverage, none of the three ever actually reached the +// mechanism: two have too little material for null-move pruning to +// be attempted at all, and the third gets resolved directly off the +// local tablebase before search ever touches null-move logic. That +// made the internal-counter approach fragile and environment- +// dependent in a way a real best-move suite isn't. See +// tests/zugzwang.epd (run with the `epd` command, e.g. `epd +// tests/zugzwang.epd`) instead -- five known zugzwang positions +// checked by best-move outcome, which doesn't care why the engine +// gets it right, just whether it does. +// + +// +// QSearch is meant to stay small and shallow (search.c's own +// ASSERT(pf->uQsearchNodes < 20000) at its only call site is a +// no-op in a release build -- this makes a version of that bound a +// real, always-checked test instead of a DEBUG-only assertion). +// +// Note this checks ctx->sCounters.tree.u64TotalNodeCount (whole +// search, root.c's own unconditional per-node ++), not +// u64QNodeCount specifically -- that counter, and every other field +// in sCounters, is wrapped in the INC() macro (chess.h), which +// compiles to nothing unless PERF_COUNTERS is defined, and this +// self-test suite is built with plain `gmake TEST=1` (see CLAUDE.md), +// which doesn't define it (confirmed live: a real `sd 3; go` on this +// TEST-only binary reports "0 qnodes" no matter how much qsearch +// activity actually happened). A whole-search bound is a strictly +// weaker signal than a qsearch-only one -- it wouldn't distinguish +// "qsearch exploded" from "full-width search exploded" -- but it's +// the honest thing this build can actually check; rebuilding with +// PERF_COUNTERS added would be needed to test u64QNodeCount for real. +// +// "Small" is also exactly what makes qsearch trees hand-auditable in +// principle: a person could Trace() every node visited on one of the +// positions below and read the resulting tree start to finish against +// the board, unlike a full alpha-beta tree at real search depth. No +// such tracing is wired up here (that would need a new instrumented +// build define, not written this session) -- noting it as a +// worthwhile follow-up, not implying it already exists. +// +FLAG +TestQSearchSanity(void) +{ + static const char *rgszFen[] = + { + // Quiet position -- qsearch should do almost nothing (stand + // pat immediately, no captures available). + "4k3/8/8/8/8/8/8/4K3 w - - 0 1", + // A short, bounded capture sequence on one square. + "4k3/8/8/3p4/2P5/8/8/4K3 w - - 0 1", + // A deeper capturing melee -- the kind of position qsearch + // exists for. + "r1bq1rk1/pp2bppp/2n1pn2/2pp4/3P4/2NBPN2/PP3PPP/R1BQ1RK1 w - - 0 1", + "r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 1", + }; + const ULONG uDepth = 3; + SEARCHER_THREAD_CONTEXT *ctx; + POSITION pos; + SCORE iScore; + ULONG u; + FLAG fRet = TRUE; + + Trace("Testing QSearch sanity (node-count bound, termination)...\n"); + ctx = SystemAllocateMemory(sizeof(SEARCHER_THREAD_CONTEXT)); + ASSERT(ctx); + + for (u = 0; u < ARRAY_LENGTH(rgszFen); u++) + { + if (FALSE == FenToPosition(&pos, (char *)rgszFen[u])) + { + UtilPanic(INCONSISTENT_STATE, + NULL, NULL, NULL, NULL, + __FILE__, __LINE__); + } + // uMaxDepth == 3 plies -- enough that the last full-width ply + // (uDepth == 0 in Search()'s ONE_PLY units) actually falls + // below Search()'s THREE_QUARTERS_PLY qsearch-entry threshold + // and real qsearch nodes get visited (measured live: depth 1 + // resolved every position here without qsearch ever firing -- + // g_uQNodeCount stayed 0 -- since a two-king or short-capture + // position this shallow settles via CommonSearchInit's own + // early-outs before the qsearch dispatch is ever reached). + // Still small enough to trace by hand per this function's own + // header comment. + ClearHashTable(); + DirtyHashTable(); + iScore = _FixedDepthSearchScore(ctx, &pos, uDepth); + + if ((INVALID_SCORE != iScore) && + !IS_VALID_SCORE(iScore)) + { + Trace("TestQSearchSanity: position %lu returned an " + "invalid score %d -- FEN: %s\n", u, iScore, + rgszFen[u]); + fRet = FALSE; + } + Trace("TestQSearchSanity: position %lu total nodes this " + "search: %llu\n", u, + ctx->sCounters.tree.u64TotalNodeCount); + + // Not the same invariant as search.c's per-entry + // ASSERT(pf->uQsearchNodes < 20000) (that resets per qsearch + // entry point and counts qsearch nodes only; this is a + // whole-search cumulative total, per this function's header + // comment on why), but a real, always-checked (not DEBUG-only) + // ceiling on "a shallow fixed-depth search on a small position + // stays small" -- a qsearch that's stopped terminating cleanly + // (e.g. a check-extension/repetition interaction letting + // QPLIES_OF_NON_CAPTURE_CHECKS get bypassed) will blow through + // this by orders of magnitude, not marginally, regardless of + // it being mixed in with full-width nodes here. + if (ctx->sCounters.tree.u64TotalNodeCount > 500000) + { + Trace("TestQSearchSanity: position %lu total node count " + "%llu is wildly outside the expected 'small' range " + "for depth %lu -- FEN: %s\n", u, + ctx->sCounters.tree.u64TotalNodeCount, uDepth, + rgszFen[u]); + fRet = FALSE; + } + } + + if (FALSE == fRet) + { + UtilPanic(TESTCASE_FAILURE, + NULL, "TestQSearchSanity", NULL, NULL, + __FILE__, __LINE__); + } + ClearHashTable(); + SystemFreeMemory(ctx); + return(fRet); +} #endif |
