path: root/src/docs/TESTING_PLAN.md

# Testing plan: closing coverage gaps in test*.c

Written 2026-09-16 after an audit of every `src/test*.c` file and how
(or whether) each is wired into `main.c`'s self-test sequence. Goal
per Scott: not more test code for its own sake -- find the
years-old bugs that asserts have been missing. Priorities: search,
eval, generate (not exclusively). Perf testing is also explicitly
in scope.

## Dead/disconnected infrastructure (fix first -- near-zero cost)

Several tests already exist but don't run. Recovering this coverage is
strictly cheaper than writing anything new, so do this before anything
else below.

1. **SEE has zero active correctness testing.** `testsee.c`'s own
   header comment admits it: `DebugSEE`/`DebugSEERecursive` (a
   brute-force make/unmake exchange-walk oracle meant to check `SEE()`
   against) is wrapped in `#ifdef TEST_BROKEN` -- a define nothing
   sets. `testsee.c`'s only *called* functions (`TestGetAttacks`,
   `TestIsAttackedBB`) test attack generation, not SEE. Given SEE is
   what `GetMoveSee`/extensions/QSearch pruning all trust (and is the
   exact code reworked in commit c0ac3d7), this is the single
   highest-value fix here. Get `TEST_BROKEN` -> live (`TEST`), fix
   whatever bitrot surfaces, and confirm it's called from `main.c`.
2. **`TestEvalWithKnownPositions`** (`testeval.c`, ~20 hand-curated
   "side X should be better here" positions from chess literature) is
   defined but **never called from anywhere** -- not `main.c`, not
   `command.c`. Wire it into the self-test sequence.
3. **`TestRecognExhaustiveKNKP`** (`testrecogn.c`, exhaustive KNKP
   endgame-recognizer enumeration) is defined and prototyped in
   `chess.h` but referenced only in a comment -- never invoked. Only
   the smaller table-driven `TestRecogn` runs. Wire it in.
4. **The hash table has no correctness self-test at all.**
   `testhash.c` contains `SanityCheckHashEntry` (an entry-shape
   validator) and `AnalyzeFullHashTable` -- but the latter is a
   post-search *diagnostic dump* called from `root.c`, not a test, and
   it's never invoked from the `TEST` self-test sequence. See "New
   test: hash correctness" below for what should replace this gap, not
   just a wiring fix.
5. **`TestEvalWithSymmetry`** only runs a *single* random symmetric
   position per invocation, and only when a human types `eval` at the
   prompt in an `EVAL_DUMP` build (`command.c:2203`) -- not part of the
   automated suite, never loops. See "New test: eval symmetry fuzz"
   below.

## New tests, by area

### Search (thinnest area relative to its bug surface)

- `TestSearch` (`testsearch.c`) runs 20 random positions at fixed
  **depth 2** only, checking solely "returned move is legal" and
  "didn't blow the time limit."
  - **DONE -- hash-consistency test** (`TestSearchHashConsistency`,
    testsearch.c): searches a position to depth D twice in a row with
    a warm hash table (scores must match), and separately confirms a
    depth-D2 search seeded by a shallower depth-D1 warm-up agrees with
    a from-scratch depth-D2 search. Wired into the `TEST=1` self-test
    sequence (main.c).
  - **DONE (partially) -- QSearch sanity test**
    (`TestQSearchSanity`, testsearch.c): checks search termination
    (no `INVALID_SCORE`/garbage) and a whole-search node-count
    ceiling. Caveat found and documented in the function's own header
    comment: the self-test suite builds with plain `gmake TEST=1`,
    which doesn't define `PERF_COUNTERS` -- every `sCounters` field
    (including `u64QNodeCount`, the qsearch-only counter this was
    originally meant to check) is wrapped in `chess.h`'s `INC()` macro,
    a no-op without that define. Confirmed live (`sd 3; go` on a
    `TEST`-only binary reports "0 qnodes" regardless of real qsearch
    activity). The test currently checks
    `ctx->sCounters.tree.u64TotalNodeCount` instead (root.c's
    unconditional per-node `++`, always live) -- a real but strictly
    weaker signal, since it can't distinguish "qsearch exploded" from
    "full-width search exploded." Building a `PERF_COUNTERS`-enabled
    `TEST` profile (or adding an always-live qsearch-only counter) is
    the natural follow-up to close this for real. The hand-auditable-
    tree idea (`Trace()` every qsearch node on a small position) is
    still worth doing but wasn't implemented this session.
  - **NOT DONE -- determinism test** (same position/depth searched
    twice, PV must match): explicitly descoped -- Scott's call is that
    a stable-PV requirement makes the test break every time move
    ordering changes for unrelated reasons, which is just noise, not a
    real regression signal. A same-*score* (not same-PV) check is
    already covered by the hash-consistency test above.
  - **NOT DONE -- MP-vs-single-thread agreement test**: still open.
    `split.c` has no dedicated correctness test.
  - **DONE (as an EPD suite, not a C test) -- null-move/zugzwang
    soundness**: `tests/zugzwang.epd` (5 positions, run via the new
    `epd` command -- see below) checks actual best-move outcome
    instead of an internal counter. An earlier attempt asserted
    against `ctx->sCounters.extension.uZugzwang` (the verification-
    search hit counter) using the 3 classic literature zugzwang
    positions, but none of them structurally reach that code path on
    this build: 2 have too little material for
    `WeShouldTryNullmovePruning`'s own `> 2` non-pawn-count gate to
    attempt null-move pruning at all, and the 3rd (despite landing in
    the eligible window) gets resolved directly off this box's local
    Syzygy tablebase before search ever touches null-move logic --
    confirmed live via the `TABLEBASE MOVE` trace line. Removed in
    favor of the EPD suite, which doesn't care about internal
    mechanism, only the final answer. See `testsearch.c`'s comment
    where `TestNullMoveZugzwang` used to be for the full writeup.

**New infrastructure that came out of chasing the above (search.c
touches none of this, but it's now how any future EPD-based search
test should be added):**

- **`epd <file>` engine command** (`EpdCommand`, script.c) parses and
  runs a raw EPD suite file directly -- no perl preprocessing step
  first. Backed by `ParseEpdRecord` (fen.c), real per-record EPD
  parsing (4 position fields, optional FEN-style halfmove/fullmove
  counters, then `;`-separated `bm`/`am`/`id` opcodes).
- **Found and fixed while building this**: `tests/convert_epd.pl` (the
  old perl-based EPD-to-script converter) had a real, silent-
  corruption bug -- its `bm`/`am` regex used `[^b]+` as the filler
  between the board and the opcode, which breaks on *every*
  Black-to-move record (the side-to-move field is literally the
  character `b`), and since `$fen`/`$id`/`$solution` were unscoped
  globals rather than per-line locals, a failed match silently carried
  the *previous* record's values forward instead of erroring.
  Confirmed live: feeding it a 5-position batch with one Black-to-move
  entry produced a byte-for-byte duplicate of the previous record for
  that one line, no diagnostic at all. The script is left in place
  (marked superseded in its own header) in case something still shells
  out to it, but new suites should use `epd` instead.
  Cross-checked the new parser against the old pipeline on
  `ecm_ringers`: `script` (old `.ep_` format) and `epd` (new
  `tests/ecm_ringers.epd`, produced by `ep_to_epd.pl`, see next item)
  produced byte-identical node counts and solve stats.
- **`tests/ep_to_epd.pl`**: one-off reverse converter, turns the
  existing `setboard`/`solution`/`avoid`/`id`/`go` script-format
  suites back into real EPD now that `epd` can read it directly. Run
  against every genuine tactical suite in `tests/` (not `bench.ep_` or
  `trial.ep_`, which use `sd`/`st`/`easy`/`quit` and aren't EPD
  underneath) -- produced `.epd` siblings for `arasan`,
  `ecm_confident_quick`, `ecm_confident_sample2`, `ecm_confident`,
  `ecm_hard_quick`, `ecm_hard`, `ecm_late`, `ecm_quick`, `ecm_ringers`,
  `ecm-gcp`, `ecm-hard`, `ecm`, `ecm016_only`, `LCT2`, `mes400`,
  `newiq`, `quick`, `swiss`, `wac`. `ecm.epd` came out to 879 records,
  matching the `606/879`/`622/879` solve-count baselines already
  quoted elsewhere in CLAUDE.md.

### Eval

- Automate `TestEvalWithSymmetry` into a real fuzz loop (N random
  symmetric positions, assert `Eval() == 0` for all, as part of the
  standard self-test run) instead of the current single-shot
  manual-only invocation.
- **Adversarial/degenerate-position term-bounds test**: 0-pawn
  positions, 9-queen positions, fully-open-board positions -- extend
  `GenerateRandomLegalPosition` or hand-build cases, assert no eval
  term produces an out-of-range score (reuse the existing
  `EvalTrace`/`EVAL_DUMP` term-tracing machinery for this).
- Audit for any **incrementally-maintained eval state** (pawn hash,
  king-safety cache) and add a from-scratch-recompute-vs-incremental
  consistency check if any exists -- classic "works until the hash
  collides" bug class.
- **Eval speed regression test**: no benchmark currently exists for
  eval itself despite it being the named NPS bottleneck vs. Crafty
  (see CLAUDE.md). Same interleaved-comparison shape as the existing
  `TestGenerate*Speed` benchmarks, broken down by game phase.

### Generate (best-covered area, but gaps remain)

- Perft (`TestMoveGenerator`) only checks to **ply 4** even though
  `uLeaves[]` already has reference values to ply 6/7 sitting unused --
  extend to a `--perft-deep` slow path, and add perft-**divide**
  (per-root-move leaf breakdown) to localize failures to a specific
  move instead of just tripping a panic.
- Add perft positions specifically chosen to stress **en passant,
  underpromotion, and castling-through-check** individually (the
  current 5 FENs give general coverage but weren't chosen for this).
- `TestGenerateAllMovesSpeed` and the per-piece-type speed tests are
  **pure microbenchmarks with no correctness assertion** -- they print
  cycles and never diff the move *sets* produced by the mailbox vs.
  bitboard generators being compared. Add a sorted-move-list-equality
  assertion alongside each (both generators already run back-to-back
  in the same loop, so this is nearly free) -- otherwise a bitboard
  generator silently missing a move class just prints a suspiciously
  fast number and passes.
- `GetMoveSee`'s new non-pawn-capture SEE path (c0ac3d7) has no
  dedicated test yet -- covered by the SEE oracle fix above once live.

### Move/Position plumbing

- Reasonably solid already (`TestMakeUnmakeMove`, `TestExposesCheck`,
  `TestIsAttacked`). Add a **long-game fuzz test**: play N random games
  of ~200 plies each (not just 1-ply-at-a-time exercises), asserting
  `VerifyPositionConsistency` and `ComputeSig`-from-scratch agreement
  with the incrementally-maintained signature at every ply. Confirm
  `VerifyPositionConsistency` is actually compiled live in the
  `TEST`/`DEBUG` build, not just present in source.

### Hash (currently zero correctness coverage)

New test needed, not just a wiring fix:
- Store/probe roundtrip correctness.
- Replacement-scheme behavior under forced collisions (fill table past
  capacity with distinct signatures, confirm replacement policy
  behaves as designed, not just "doesn't crash").
- `ADJUST_MATE_IN_N` ply-distance adjustment on `HASH_FLAG_EXACT`
  entries specifically -- the exact mechanism CLAUDE.md flags as
  audited-by-hand but not test-covered after this session's `ASSERT`
  removals in `QSearch()`.

### Recognizer

- Wire in `TestRecognExhaustiveKNKP` (see dead-infra list above).
  Consider the same exhaustive-enumeration treatment for any other
  small recognized endgame class in `recogn.c` (KPK, KRK) if one
  exists without a test.

## Perf testing

Current coverage is narrow: five isolated per-piece-type
mailbox-vs-bitboard microbenchmarks (`TestGenerate*Speed`), nothing for
search, eval, or hash.

- **Eval NPS/cycles-per-call benchmark**, same interleaved-comparison
  shape as the existing generate benchmarks, by game phase -- eval is
  the named bottleneck (CLAUDE.md's 1.5M vs 7.5M nps gap vs. Crafty)
  and has no dedicated speed test today.
- **Whole-search NPS benchmark**: `bench.c` already exists for this --
  confirm whether it's part of the automated `TEST` self-test pass or
  purely a manual `bench` command; if manual-only, wire a
  fixed-node-budget version into the automated sequence so a
  regression surfaces without someone remembering to run it by hand.
- **Hash-probe-under-load benchmark**: cycles/probe at low occupancy
  vs. near-full table -- replacement-scheme cost can regress silently
  today with nothing to measure it.
- **Stored baselines with pass/fail thresholds**, not print-only
  output: today every perf test prints cycles/call and leaves
  interpretation to a human reading the log. Store a last-known-good
  baseline (same idea as `head_reference/` for solve-counts) and fail
  loud if a call site regresses past some tolerance.

## Suggested priority order

1. Un-bitrot the three dead tests (SEE oracle,
   `TestEvalWithKnownPositions`, `TestRecognExhaustiveKNKP`) --
   near-zero cost, immediate coverage recovery.
2. Hash-table correctness test (store/probe/replace/mate-adjust) --
   currently the single biggest correctness blind spot.
3. Search determinism + MP-vs-single-thread agreement test -- targets
   the exact class of bug most likely to hide in threading/hash/
   extension interactions that per-function unit tests can't see.
4. Extend perft to deeper plies + divide output; add move-set-equality
   assertions to the existing generate speed benchmarks (turns
   dead-weight microbenchmarks into correctness tests for free).
5. Eval: automate the symmetry fuzz test (loop N, not 1) and add
   adversarial/degenerate-position term-bounds checks.
6. Round out perf testing per the list above, with stored baselines
   and pass/fail thresholds instead of print-only output.

## "Fuzz" testing

**Implemented: `fuzz_test.sh [binary] [depth] [batch_size] [cpus]`**
(defaults: `./_typhoon`, `sd 6`, 200 positions/batch, `--cpus 2`). Loops
forever: each iteration draws a fresh random batch of `setboard`/`go`
records from the full 400,000-line / 200k-position
`tests/cross_sectional_200k.ep_` pool, runs them through a `DEBUG=1`
build under gdb, and scans the gdb output for a crash/assert signature
before starting the next batch. Stops (exit 1) and prints the offending
batch + engine log paths as soon as one is found; otherwise runs
indefinitely (`^C` to stop). Logs land under `/usr/local/tmp/typhoon/fuzz/`
(one sampled `.ep_` batch + one `.gdb.txt` + one engine `--logfile` per
iteration).

**Two build/invocation bugs found and fixed getting this working, neither
a sandbox limitation despite first appearances:**

1. **The DEBUG binary must be built with `MP=1` as well as `DEBUG=1`**
   (`gmake clean; gmake DEBUG=1 MP=1`) -- `--cpus` is compiled in only
   under `#ifdef MP` (`main.c:311`). A `DEBUG=1`-only build silently
   treats `--cpus` and its numeric argument as two separate unrecognized
   arguments (`main.c:400`'s "Error (unknown argument)" path) and falls
   back to the 1-cpu default with no hard failure -- easy to miss, and it
   also means `split.c`'s parallel-search code never actually links in,
   defeating the point of passing `--cpus 2` at all.
2. **`gdb -batch -ex "handle SIGTRAP nostop noprint pass" -ex run ...`
   breaks thread control against this MP/pthread build specifically**,
   producing `ptrace (PT_SUSPEND): Operation not permitted` followed by
   `Child process unexpectedly missing: No child processes` -- looked
   exactly like a sandboxed environment blocking ptrace outright (and was
   initially misdiagnosed as that), but reproduced identically outside
   any harness/sandbox, and disappeared entirely once that one `-ex` line
   was dropped. `fuzz_test.sh` omits it; a plain `run`/`bt full`/
   `info registers` gdb batch invocation debugs the MP binary fine,
   including catching a genuine `int3` breakpoint from a failed `ASSERT`.

Not yet done: no crash has been found by a run so far, so the
crash-detection path (the `grep -qE` in `fuzz_test.sh`) is exercised by
its own logic reading correctly, not yet by a real hit. Worth periodically
running this for stretches (it's unbounded/`^C`-driven, unlike the other
gates) rather than treating it as a one-time pass/fail check like
`debug_smoke_test.sh`. See also: `./debug_smoke_test.sh` (fast, curated
sample, single pass/fail) for the pre-commit-sized version of the same
idea.