diff options
| author | Scott Gasch <[email protected]> | 2026-10-15 11:26:31 -0700 |
|---|---|---|
| committer | Scott Gasch <[email protected]> | 2026-10-15 11:26:31 -0700 |
| commit | 53b380c658f828c4c0e096be2763e6b012c23abd (patch) | |
| tree | 3e6b6a8ce608674d8fa33e6e9632d4d37290a301 /src | |
| parent | ef2c2f667c4fc2c0303f56ea487183916f82fb67 (diff) | |
Docs moved around.
Diffstat (limited to 'src')
| -rw-r--r-- | src/ANALYZE_GAME.md | 254 | ||||
| -rw-r--r-- | src/CLAUDE.md | 74 | ||||
| -rw-r--r-- | src/PROFILE.md | 222 | ||||
| -rw-r--r-- | src/TESTING_PLAN.md | 298 |
4 files changed, 21 insertions, 827 deletions
diff --git a/src/ANALYZE_GAME.md b/src/ANALYZE_GAME.md deleted file mode 100644 index 2576a1e..0000000 --- a/src/ANALYZE_GAME.md +++ /dev/null @@ -1,254 +0,0 @@ -# Analyzing a game from blunders.pgn - -A working playbook for pulling a specific game out of `blunders.pgn` and -figuring out *where the eval/search actually went wrong*, as opposed to -just noticing that the final result was bad. Written after a session spent -doing exactly this on the 2026-10-06 `typhoon vs Crafty-25.2` loss (a -KRP-vs-KQ endgame that looked like it had a scoring bug but turned out to -be legitimately lost, plus two real engine bugs found along the way). - -## 0. What's actually in blunders.pgn - -Each game's move comments are **only** `{score/depth time}`, e.g. -`{-5.09/34 6}` -- score in pawns from White's perspective (not side-to-move; -confirmed by the sign trending consistently negative as White loses, across -both White's and Black's own move comments), search depth reached, and -wall-clock seconds spent on that move. **There is no PV text and no engine -tag in the PGN at all** -- not `<REC>`, not `<EGTB>`, not `<#>`, nothing. -Whatever wrapper plays these self-play/vs-Crafty games strips the PV down to -just the eval number before writing the comment. If you want to see *why* -a move was chosen -- the actual PV, or whether a recognizer/tablebase hit -produced the score -- you have to replay the position through a live engine -yourself; the PGN alone can't tell you. - -Don't go looking for `<EGTB>`/`<#>` tags in old PGNs for another reason too: -those sentinels (`EGTBMOVE` plus the `RECOGN_EGTB` wiring) are new, added in -the 2026-10-06 session that produced this document. Any game recorded before -that commit landed couldn't have produced them even if the PGN format did -carry PV text. - -## 1. Pulling the game and exact FENs with python-chess - -`python3 -c "import chess"` works in this environment (python-chess is -installed) -- use it rather than hand-parsing SAN, especially for games deep -enough that move numbers and ply counts diverge from intuition. - -```python -import chess.pgn -f = open('blunders.pgn') -games = [] -g = chess.pgn.read_game(f) -while g: - games.append(g) - g = chess.pgn.read_game(f) -last = games[-1] # or games[N] for a specific game -board = last.board() -moves = list(last.mainline_moves()) -``` - -**Getting a FEN at a specific PGN move number**: ply count and PGN move -number are NOT the same thing and it's easy to get this wrong (did exactly -this once already this session -- cost a wasted round-trip). The mapping: - -```python -def ply(movenum, color): # color: 'w' or 'b' - return 2*(movenum-1) + (1 if color == 'w' else 2) -``` - -`moves[ply(69, 'w') - 1]` is White's 69th move (0-indexed list, 1-indexed -ply). Push moves up through the target ply and read `board.fen()` -- -`board.halfmove_clock` is directly available too and matters a lot (see -"fifty-move-rule proximity" below). - -To scan a whole stretch and watch halfmove_clock / material change move by -move (useful for finding exactly where a pawn/capture resets the fifty-move -counter, or where material transitions from one tablebase signature to -another): - -```python -for i, m in enumerate(moves, start=1): - before = board.copy() - board.push(m) - if lo <= i <= hi: - print(i, before.san(m), 'halfmove_clock=', board.halfmove_clock, - 'fen=', board.fen()) -``` - -## 2. Replaying a position through typhoon - -Use `--batch --command` with `setboard`, not interactive stdin piping (per -CLAUDE.md, piping commands over stdin to a live REPL doesn't reliably -complete a multi-step script). Always pass `--egtbpath` explicitly if you -want to be sure which EGTB directory is in play (the hardcoded default is -`/zscratch/egtb`, which currently has full WDL+DTZ coverage through 6-man -- -see CLAUDE.md -- but don't assume that silently when writing up findings; -state it). - -```sh -./typhoon --cpus 1 --hash 256m --egtbpath /zscratch/egtb \ - --logfile /tmp/typhoon/<name>.log \ - --batch --command "force; setboard <fen>; eval; sd <depth>; go" -``` - -- `eval` alone (no search) gives the raw static eval -- useful as a sanity - baseline before trusting a search score. -- `sd <depth>` fixes search depth; prefer this over `st <seconds>` for - reproducibility (machine load varies a lot on this shared box -- see - CLAUDE.md's ECM section). -- Watch stdout for these diagnostic lines, which tell you *how* the move was - chosen, not just what it was: - - `ONLY MOVE --> stop searching now` -- forced, no real search happened. - - `TABLEBASE MOVE --> stop searching now` -- root EGTB (DTZ) root-move - short-circuit; score will show as `PV=<EGTB>` and a flat `±NMATE`-ish - "mate" display that is **not** a real mate distance (see section 4). - - `FORCED MATE --> stop searching now` -- the iterative loop detected a - mate score at the current depth and is stopping; the printed PV comes - from `szLastPV`, which on a **root fail-high** is just the bare move - (`MoveToSan`), not a real PV walk -- see section 5, don't expect `<#>` - here. - - `DEPTH LIMIT --> stop searching now` -- ordinary completion, nothing - special. -- Grep the output/log for `tellothers` to get the final - `score, time, nps, PV=...` summary line in one place. - -## 3. Cross-checking with Crafty - -`/usr/local/bin/crafty` is available and gives a genuinely independent -second opinion -- different engine, different (or absent) EGTB usage, -different search. Use it exactly the way you'd use typhoon: - -```sh -echo "setboard <fen> -sd <depth> -go -quit" | timeout 20 /usr/local/bin/crafty 2>&1 | tail -30 -``` - -Crafty's output is far chattier (full eval component breakdown, PV with -move count, nodes/nps) -- look for the `depth ... score ... variation` -block and the `note: scores are for the white side` line (confirms sign -convention, which is directly comparable to typhoon's White-POV scores). - -**Why this matters**: if typhoon reports something wild (a huge mate-ish -score with a PV that goes nowhere, or a score frozen for 20+ plies in a row) -and Crafty reports an ordinary, non-mate score with a PV that's actually -converging, that's your signal the *typhoon* number is suspect, not that the -position itself is unusual. This exact comparison is what caught the -`ProbeEGTBRoot` DTZ-as-mate-distance bug (section 4) -- typhoon said -`-MATE4`, Crafty's independent 16-ply search said `-7.38` with no mate -anywhere in a 13-move PV. - -## 4. Things that look like bugs but are fifty-move-rule artifacts (and one that was a real bug) - -**Watch `halfmove_clock` whenever a long endgame grind produces a -suspicious score.** Syzygy tables report **WDL** (win/draw/loss) and -**DTZ** (distance to the next fifty-move-rule-*zeroing* move -- a capture or -pawn push), not DTM (distance to mate). This project's EGTB directory has -**zero `.rtbm` files** -- DTM tables were never downloaded, and Fathom's DTM -code (`probe_dtm_table`, `TB_probe_dtm`, etc.) is present in the source but -dead weight without data behind it. So: - -- A genuinely won endgame can sit at a **small DTZ** while still being many - plies from actual mate, especially once `halfmove_clock` is climbing - toward 100 (a reset becomes forced soon regardless of how far mate - actually is). -- A score that stays flat for many consecutive moves, or a score that - jumps to a huge "mate-ish" magnitude with a PV that doesn't resolve - anywhere close to that many plies later, is the signature of this -- - **not** evidence of missing/corrupt tablebase files by itself. Check - `ls /zscratch/egtb | grep <material-signature>` to rule out a genuine - coverage gap first, but if the files are present, suspect the - DTZ-vs-mate-distance conflation next. -- This was a **real, fixed bug** as of 2026-10-06: `probe.c`'s - `ProbeEGTBRoot` used to compute `*piScore = INFINITY_SCORE ∓ uDtz` and - report it through the mate-score display path -- i.e. it fabricated a - "mate in N" number out of a quantity that was never a mate distance. Fixed - to report a flat `+NMATE`/`-NMATE` (same convention `ProbeEGTB`/ - `RecognLookup` already use at interior nodes) instead. If you're - diagnosing a *pre-fix* game (anything before this fix landed -- check - `git log probe.c`), expect to see exactly this artifact and don't chase - it as a new finding. -- The **move** Fathom picks at the root is unaffected by any of this -- - `tb_probe_root()` does its own internal DTZ-based ranking before - returning, so the chosen move is already DTZ-optimal regardless of - whether the *score* we derived from it was sane. Don't second-guess the - move choice because the displayed score looked wrong; check the score - logic and the move logic as separate questions. - -## 5. Reading PV sentinels (and when you won't see them) - -`WalkPV` (`util.c`) terminates a printed PV early and appends a tag when it -hits one of five sentinel "moves" (`data.c`): `HASHMOVE` → `<TT>`, -`RECOGNMOVE` → `<REC>`, `DRAWMOVE` → `<=>`, `MATEMOVE` → `<#>`, `EGTBMOVE` → -`<EGTB>`. These mark *why the PV stopped*, not what the move was: - -- `<REC>` / `<EGTB>`: an interior-node `RecognLookup` hit resolved the - score outright. `RECOGN_EGTB` (→ `<EGTB>`) means an actual on-disk Syzygy - WDL probe succeeded; `RECOGN_EXACT` (→ `<REC>`) means one of `recogn.c`'s - hand-written classic recognizers (KPK, KBKP, etc.) fired instead. Before - 2026-10-06 these were indistinguishable in the PV (both reported - `RECOGNMOVE`) -- `RecognLookup` had an `RECOGN_EGTB` enum value sitting - unused the whole time, always returning `RECOGN_EXACT` even on a genuine - Fathom hit. If you're looking at an old log/build, `<REC>` could mean - either; `<EGTB>` specifically didn't exist yet. -- `<#>`: search found an actual forced checkmate (zero legal moves + in - check) at a PV leaf. -- `<=>`: a drawn leaf (zero legal moves, not in check == stalemate; this - sentinel is also used for recognized/forced draws elsewhere). -- **Root tablebase moves get `<EGTB>` in `szLastPV` directly** (as a literal - string, not via `WalkPV` -- `ProbeEGTBRoot`'s success path short-circuits - the whole iterative loop and never calls `WalkPV` at all for that move). - -**The big gotcha**: `PrintPV` (`util.c`) only calls the real `WalkPV` walk --- the one that can surface any of these tags -- when the reported score is -a genuine PV node, `iAlpha < iScore < iBeta`. On a **root fail-high** -(`iScore > iBeta`, i.e. the aspiration window is still widening upward), -it falls back to printing just the bare best move via `MoveToSan`, with no -PV walk and therefore no sentinel, ever -- even if the position is a -dead-obvious mate-in-1 whose leaf genuinely wrote `MATEMOVE` two plies down. -So: **don't read "no `<#>` in the output" as "no mate was found"** -- check -whether the line says `FORCED MATE` (a fail-high-driven stop) vs. a normal -depth-complete report before concluding anything about whether the engine -actually resolved the mating line as an exact PV. - -## 6. The timer-elapsed-time gotcha - -If you see `Searched for -<huge number> seconds` in output, that's not a -real timing anomaly to chase -- or rather, it *was* a real bug (fixed -2026-10-06): `Iterate()` (`root.c`) used to stamp -`g_MoveTimer.dEndTime = SystemTimeStamp()` partway through the function, -*before* the shared `end:` label that every early-exit path (`ONLY MOVE`, -`TABLEBASE MOVE`, pre-move terminal states) jumps to via `goto end`. Those -paths skipped the stamp entirely, leaving `dEndTime` stale (0, or whatever -the previous unrelated search last set it to), which produces exactly this -kind of nonsense negative "elapsed time" when the end-of-search report -computes `dEndTime - dStartTime`. Fixed by moving the stamp to right after -the `end:` label itself. If you're diagnosing a pre-fix build and see this, -it's cosmetic (doesn't affect move choice or score), but it's also a tell -that you're looking at a root short-circuit path (tablebase or only-move), -which is useful context for *why* no real iterative search happened at that -node. - -## 7. General workflow summary - -1. Identify the game (usually "last game in the file") and get the exact - FEN at the move(s) you care about via python-chess -- don't eyeball SAN - move numbers against ply counts by hand. -2. Check `board.halfmove_clock` at that FEN before anything else if it's an - endgame -- rules out/in the fifty-move-rule class of artifact immediately. -3. Replay the FEN through `./typhoon --batch --command "force; setboard - <fen>; sd <depth>; go"`, read the diagnostic `--> stop searching now` - line to understand *which* code path produced the move, and check - `tellothers`'s `PV=` field. -4. Get a second opinion from `/usr/local/bin/crafty` on the same FEN at - comparable depth. Large disagreement (especially "my engine claims near- - mate, Crafty's PV goes nowhere near that far") is the signal to dig into - the score-computation code, not the position. -5. If something still looks wrong, read the actual source for the code path - the diagnostic line pointed at (`root.c`'s `Iterate`, `probe.c`'s - `ProbeEGTB`/`ProbeEGTBRoot`, `recogn.c`'s `RecognLookup`) rather than - guessing from behavior alone -- every bug found this way so far - (`ProbeEGTBRoot`'s DTZ-as-mate-score, the `dEndTime` placement, the - dead `RECOGN_EGTB` branch) was a straightforward, small, findable-in- - minutes fix once the actual code was in front of us; none of them - required deep speculation. diff --git a/src/CLAUDE.md b/src/CLAUDE.md index 0bfc028..f2bbd37 100644 --- a/src/CLAUDE.md +++ b/src/CLAUDE.md @@ -1,6 +1,6 @@ # typhoon chess engine -A chess engine by Scott Gasch ([email protected]), originally started 2004. +A chess engine by Scott Gasch ([email protected]), originally started 2000. Previously named "monsoon" -- any old references to that name mean this same codebase. Speaks the xboard/WinBoard protocol. FreeBSD host, 64-bit. @@ -26,7 +26,13 @@ exist here (no `gcc`, and the resulting `.o` files are architecture-mismatched against everything else). **Don't use build.sh as-is.** Build directly with: ```sh -gmake -j5 GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1 +gmake release PERF_COUNTERS=1 MP=1 +``` + +or + +```sh +gmake debug PERF_COUNTERS=1 MP=1 ``` This uses `clang`/`clang++` (already the GNUmakefile default) and produces a @@ -79,50 +85,6 @@ invariants) fast; the suite workflow measures whether a change is actually directory isn't left with a `TEST`/`DEBUG`-flavored binary as a side effect of running it. ~20s end to end on this box. -**Two real bugs found and fixed by exercising this session's changes -through the smoke test, both pre-existing and unrelated to the session's -actual move-ordering work:** -1. **A killer-table backfill collision** (`dynamic.c`'s `_NewKillerMove`, - from a previous session's `917bf12`): when slot `[1]` was empty after - the shift, it backfilled unconditionally from - `mvNullmoveQuietRefutations[uPly]` with no check that the backfilled - move differed from the move just placed in slot `[0]` -- if they - coincided, both slots ended up holding the identical move, tripping - `_NewKillerMove`'s own `!IS_SAME_MOVE(mvKiller[uPly][0], mvKiller[uPly][1])` - invariant. Silent (a wasted killer slot, not a crash) in a release - build, since `ASSERT` compiles to nothing there -- only visible once a - `DEBUG` build actually got exercised. Fixed by skipping the backfill - when it would collide. -2. **Two `ASSERT(iBestScore > -NMATE)` calls in `QSearch()`** (`search.c`) - encoded an invariant that isn't actually guaranteed: at a very early - (full-width) root iteration, or after aspiration-window widening - following repeated fail-highs, an ancestor frame's `iAlpha`/`iBeta` can - itself already be more extreme than `-NMATE` (e.g. `-32767`) with no - mate anywhere in the line. Once that's true, a legitimately-computed - fail-low placeholder (`iBestScore = iAlpha`, used to deny a stand-pat - credit when material looks unavoidably in trouble) or a genuine - fail-high (`iScore >= iBeta`) can land in mate-magnitude territory - purely as a window artifact -- there's no value that can simultaneously - satisfy `iBestScore <= iAlpha` (required to preserve fail-low semantics) - and `iBestScore > -NMATE` once `iAlpha` itself is past that threshold, - so the assert was demanding something structurally impossible, not - catching a real bug. Removed both. This exposure (a non-mate score - numerically resembling one) already existed in every release build - before this, since `ASSERT` is a no-op there -- checked `hash.c`'s - storage path specifically and confirmed it already treats any value - `<= -NMATE` as a sound, if loose, upper bound regardless of origin - (clamped to exactly `-NMATE` on store, never marked `HASH_FLAG_EXACT`, - so the `ADJUST_MATE_IN_N` ply-distance readjustment -- which only fires - for `HASH_FLAG_EXACT` entries -- never misapplies to it); a full audit - of every other `NMATE`/`MATED_SCORE` consumer (root.c's mate-in-N - display being the obvious remaining one) hasn't been done. - -Also fixed while chasing these: batch-mode's exit code was `exit(-1)` -(`command.c`, truncates to `255` as an 8-bit status), indistinguishable -from a real crash's nonzero exit -- changed to `exit(0)` so a clean batch -run and a crash are actually distinguishable by exit status, which is what -`debug_smoke_test.sh` relies on. - **On not making this an automatic git pre-commit hook**: deliberately not wired up that way. Two concrns pushed toward manual-by-default: (1) each full gate is a `gmake clean` + rebuild, real time on every commit even for @@ -150,6 +112,8 @@ an xboard-protocol REPL on stdin/stdout. Key commands used during development: - `evaldna dump` / `evaldna read <file>` / `evaldna write <file>` -- the eval-constant import/export system (`g_EvalDNA` in eval.c). `evaldna` alone (no subcommand) just prints usage -- you need `evaldna dump` explicitly. +- `dump eval` -- in a EVAL_DUMP=1 build (see GNUMakefile), will output each + individual eval term at the previously evaluated position. - `sd <n>` / `st <n>` -- fixed depth vs. fixed time search limits. - `go` -- search and play one move for whichever side is on move, then stop (when paired with `force` after). Repeated `go`/`force` cycles correctly @@ -216,13 +180,6 @@ touch); for large 6-man files it's a weaker guarantee -- a `ready` whether this matters in practice; if deep-probe latency spikes show up in profiling, this is the first place to look. -**Not yet done, a natural follow-up**: root-time warm-up -- proactively -touching (probing) WDL tables for material signatures reachable within a -few plies of likely trades from the root position, before the main -search starts, so `ready[WDL]` is already true for most of what the deep -search will actually encounter rather than depending on an earlier -shallow-ply probe in the *same* search to have warmed it first. - **Also worth knowing**: `ProbeEGTB` wraps the actual `tb_probe_wdl` call in a single process-wide spinlock (`g_uEgtbLock`, `probe.c`). Increasing probe frequency deep in the tree (now possible via the above) increases @@ -484,6 +441,17 @@ bake-in-if-better, repeatable indefinitely against a growing game pool. large improvement it first looked like. Always diff ECM numbers taken at the same `sd` depth. +## Running the engine in batch workloads + +Sometimes you will want to run the engine in a --batch --command +'whatever' mode to do background batch processing. When you do, run +with --cpus=1 and --hash=8m if you don't have a reason not to. The +--hash argument is because every `setboard <fen>` command memsets the +hash table to 0 so a default (256m) hash table will add latency to the +workflow. With --cpus=1, you can run up to 16 instances of the engine +on this 24 core machine and still leave enough cycles for normal +operating while the chess work continues in the background. + ## Dynamic move ordering experiments (2026-08-29) A single long session that redesigned how quiet ("leftover", i.e. diff --git a/src/PROFILE.md b/src/PROFILE.md deleted file mode 100644 index 67106e0..0000000 --- a/src/PROFILE.md +++ /dev/null @@ -1,222 +0,0 @@ -# Profiling ./typhoon - -Two environments, two different root situations. Short version: **on -Linux, use `gmake GPROF=1` + gprof(1) -- fully root-free.** On FreeBSD, -`pmcstat`'s hardware-counter sampling is also root-free *in practice* once -`hwpmc.ko` is loaded, which is the one step that does need root (and only -once per boot). - -## Linux (this box): gprof, no root required at all - -`perf` is blocked here -- `kernel.perf_event_paranoid` is `4` (stricter -than the usual 0-3 range; Ubuntu-specific), so `perf record`/`perf stat` -refuse to run for a non-root user with no CAP_PERFMON, and there's no -non-root workaround short of asking someone to lower that sysctl or grant -the capability. `valgrind`/`callgrind` and `uftrace` aren't installed and -installing them needs `apt` (root). `gprof` (part of binutils) is already -installed and needs no privilege of any kind -- it works by compiling the -binary with `-pg`, which makes every function call `mcount()` to accumulate -call-graph edge counts, plus a `SIGPROF`-interval sampler for self-time. -Nothing about that touches hardware counters or kernel permissions. - -A `GPROF=1` gmake flag has been added to `GNUmakefile` for this (folds -`-pg` into `$(PROFILE)`, so it lands in both the compile and link steps -automatically): - -```sh -gmake clean # -pg changes codegen for every TU; must rebuild from scratch -gmake -j5 GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1 GPROF=1 -``` - -Run the resulting `./typhoon` from whatever directory you want `gmon.out` -written into (it's always named exactly that, in cwd, overwritten on each -run) -- **use `--cpus 1`**, not the usual `MP=1` multi-threaded search, for -this run specifically. See the threading caveat below for why. - -```sh -mkdir -p /tmp/typhoon/profile && cd /tmp/typhoon/profile -/home/scott/typhoon/src/typhoon --cpus 1 --hash 256m --batch --command \ - "force; sd 12; script ../tests/ecm_confident_quick.ep_" -``` - -Then read it with gprof, pointing at the *same* binary that produced it -(symbol addresses are binary-specific): - -```sh -gprof -b /home/scott/typhoon/src/typhoon gmon.out > report.txt -less report.txt -``` - -`report.txt` has two sections: a **flat profile** (self time, sorted -descending -- "what's hot") first, then a **call graph** (who calls whom, -cumulative time) after. The flat profile is usually the right place to -start. Confirmed working end-to-end on this box: a `sd 14` search from the -startpos produced a flat profile correctly topped by `Eval` (16.1% self -time, 74M calls), then `Search`, `SelectBestWithHistory`, -`_GenerateAllMovesBB`, `GenerateMoves`, etc. -- a sane-looking distribution. - -**Don't forget to rebuild the normal release binary afterward** (`gmake -clean` + the standard `GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1` profile, -no `GPROF=1`), same reasoning as `precommit_check.sh`: a `-pg`-instrumented -`typhoon` sitting in the working directory as a side effect of profiling is -easy to mistake for the real release build later. - -### Threading caveat: use `--cpus 1` for gprof runs - -glibc's gprof support predates pervasive multithreading and only reliably -attributes `SIGPROF`-based self-time samples to the thread that set up -profiling (typically the one running `main()`) -- worker threads spawned -under `MP=1`'s multithreaded search can have their CPU time undercounted -or misattributed in the flat profile. Call-graph edge counts (`mcount()` -increments into global memory, not timer-based) aren't subject to this, -but to keep the whole report trustworthy, profile with `--cpus 1` and -treat multi-threaded gprof numbers as unreliable if you ever need to look -at search scaling specifically (use the FreeBSD `pmcstat` path below for -that, or `perf` if the paranoid sysctl ever gets relaxed on this box). - -### Other caveat: inlining - -`-pg` only instruments actual (non-inlined) function calls -- a function -the optimizer inlined into its caller never gets its own `mcount()` call -and shows up folded into the caller's self time, not as its own line. The -flat profile above is still real, just coarser for small/hot helper -functions than it would be with a sampling-based (hardware PMC) profiler. - -## FreeBSD (`wannabe.house`, this engine's native/original platform): pmcstat + gprof - -Verified against the live box 2026-10-05 (session had passwordless ssh to -`[email protected]`, `~/typhoon` there, a separate enlistment from this -one). **The commonly-assumed "needs root" turned out to be only partially -true**: `hwpmc.ko` must be loaded (`kldload hwpmc`, root-only, but -typically already loaded by someone else on a shared box -- check with -`kldstat | grep hwpmc` before assuming you need to ask for it) but once -it's loaded, **process-scoped sampling with `pmcstat -P` and the whole -gprof-conversion pipeline below ran successfully as a plain, non-root user -in this session** -- no `sudo` anywhere. The earlier assumption of -needing root for every run likely came from either testing with `-S` -(system-wide mode, which genuinely does require `PRIV_PMC_SYSTEM` / -root) or from the `hwpmc.ko` load itself. - -This box is AuthenticAMD (`kern.hwpmc.cpuid: AuthenticAMD-23-71-0`), so -Intel-style event names like `CPU_CLK_UNHALTED.THREAD` don't exist. List -available raw events with `pmcstat -L`; the AMD equivalent of "unhalted -cycles" is `ls_not_halted_cyc`. The portable alias `instructions` also -works; the portable alias `cycles` does **not** work for sampling (`-P`/ -`-S`) because it's backed by the TSC, which can't deliver the -overflow interrupt sampling needs -- use `ls_not_halted_cyc` (or another -real hardware event) instead, not the `cycles` alias. - -### 0. Confirm hwpmc is loaded (one-time, needs root if not already) - -```sh -kldstat | grep hwpmc # if this prints a line, skip straight to step 1 -kldload hwpmc # root-only, one-time (until next reboot) -``` - -### 1. Collect samples (process-scoped, not system-wide) - -Use capital `-P` (process-mode sampling, follows only the spawned child) --- NOT lowercase `-p` (counting mode only, incompatible with `-O`) and NOT -`-S` (system-wide: captures every process on the box, requires root, and -makes symbol resolution in step 2 slow/noisy). - -Append `,<count>` to the event name to set the sampling interval in -cycles -- the default reload count samples far too often and produces -multi-GB output files that take forever to convert. `4000000` (~every 4M -cycles) is a reasonable starting point; increase for a coarser/faster -profile, decrease for finer resolution on a short run. - -```sh -mkdir -p /tmp/typhoon/profile -pmcstat -P ls_not_halted_cyc,4000000 -O /tmp/typhoon/profile/pmc.out -- \ - ./typhoon --cpus 1 --hash 256m --batch --command \ - "force; sd 12; script ../tests/ecm_confident_quick.ep_" -``` - -No `sudo` needed for this step as long as step 0 is satisfied. - -### 2. Convert to a gmon.out-style file, per executable - -`pmcstat -R <file> -g` writes a `*.gmon` file per executable/library seen -in the samples, auto-named, into a subdirectory named after the event -(e.g. `ls_not_halted_cyc/`) under the current directory. (`-O` cannot be -combined with `-R` -- the input file is the positional arg to `-R` -itself.) A `kernel.gmon overflowed` warning is expected/harmless in -process-mode -- ignore it, we only want the typhoon binary's own file. - -```sh -pmcstat -R /tmp/typhoon/profile/pmc.out -g -# -> creates ./ls_not_halted_cyc/typhoon.gmon (plus one per shared lib) -``` - -### 3. View the report with gprof - -FreeBSD's base gprof segfaults on the raw dump produced by `-G <file>` -(a different, two-step raw-dump path) -- skip that entirely. It does -**not** segfault on the proper gmon.out-format file produced by `-g` -above: - -```sh -gprof ./typhoon ls_not_halted_cyc/typhoon.gmon > /tmp/typhoon/profile/report.txt -less /tmp/typhoon/profile/report.txt -``` - -Same two-section report as the Linux path: call graph, then flat profile -(self time, sorted descending) -- the flat profile is usually the more -useful starting point. - -### Gotchas recap - -- `-S` = system-wide sampling, requires root, resolves symbols for - unrelated processes too -- avoid. `-P` = process-scoped, no root needed - once `hwpmc.ko` is loaded -- what we want. `-p` (lowercase) = counting - mode, not sampling, incompatible with `-O`. -- Always set an explicit `,<count>` sampling interval or the output file - balloons to multi-GB and conversion takes forever. Confirmed: an - ~11-second `sd 14` single-position search at the default interval - produced a 300MB+ sample file. -- `-O` and `-R` are mutually exclusive; `-R` takes the input file - directly as its argument. -- `-G` writes a raw dump gprof segfaults on; `-g` writes a proper - gmon.out-format file gprof handles fine. Use `-g`. -- The workload needs to run long enough for `-P` sampling to actually - catch anything -- a single quick `sd 9` search (~1s) produced a fixed - ~3.8KB output file (just the exec/mapping record, zero PC samples) - regardless of the sampling interval used. A `script <suite>.ep_` run - over a whole suite (tens of seconds to minutes, e.g. `ecm_ringers.ep_` - at `sd 10`, ~25s) or a single deep `sd 14`+ search (confirmed: ~11s) is - long enough to get real samples. -- **Not every event supports `-P` sampling** (some AMD PMC events are - counting-only). Branch-misprediction events (`ex_ret_brn_misp`, - `ex_ret_cond_misp`, etc.) work fine in counting mode (`pmcstat -p - ex_ret_brn_misp -- <cmd>`, prints a single total at exit) but produce - the same fixed ~3.8KB/zero-samples output as the "too-short workload" - case even against a long run -- a real per-event limitation, not a - workload-duration problem (confirmed by running the known-good - `ls_not_halted_cyc` event against an identical command/interval/ - duration, which produced immediate real samples). Likely cause: not - every AMD PMC event supports overflow-interrupt (PMI) delivery, which - `-P`/`-S` sampling requires -- some are counting-only, and neither - `pmcstat -L` nor `pmccontrol -L` distinguish which is which. The only - way to find out is to try `-P` against a known-long-enough workload and - check the output file size. If a per-function mispredict breakdown is - ever actually needed, look into whether a PEBS-like interrupt-capable - event exists for this, before assuming it's simply not possible. - -## Which to reach for - -- **Linux, day-to-day**: `gmake GPROF=1` + gprof. No privilege - dependency, no sysadmin step, works today. -- **FreeBSD (`wannabe.house`)**: `pmcstat -P` + gprof, after confirming - `hwpmc.ko` is loaded. Hardware-counter-based (actual cycles/ - instructions, not call-count-overhead-inflated like `-pg`), so more - trustworthy for small/hot inlined-adjacent functions, and not subject - to the multithreading self-time caveat gprof's `SIGPROF` approach has - on Linux -- but still process-scoped, so still only reflects the one - profiled process, not the whole system. -- **If Linux `perf` access ever gets relaxed** (someone lowers - `kernel.perf_event_paranoid` or grants `CAP_PERFMON`), it would be the - best of both: hardware counters *and* correct multithreaded attribution, - without `-pg`'s inlining blind spot. Worth revisiting if a future - profiling question specifically needs multi-threaded search behavior - under `MP=1`. diff --git a/src/TESTING_PLAN.md b/src/TESTING_PLAN.md deleted file mode 100644 index 7d2bfc2..0000000 --- a/src/TESTING_PLAN.md +++ /dev/null @@ -1,298 +0,0 @@ -# Testing plan: closing coverage gaps in test*.c - -Written 2026-09-16 after an audit of every `src/test*.c` file and how -(or whether) each is wired into `main.c`'s self-test sequence. Goal -per Scott: not more test code for its own sake -- find the -years-old bugs that asserts have been missing. Priorities: search, -eval, generate (not exclusively). Perf testing is also explicitly -in scope. - -## Dead/disconnected infrastructure (fix first -- near-zero cost) - -Several tests already exist but don't run. Recovering this coverage is -strictly cheaper than writing anything new, so do this before anything -else below. - -1. **SEE has zero active correctness testing.** `testsee.c`'s own - header comment admits it: `DebugSEE`/`DebugSEERecursive` (a - brute-force make/unmake exchange-walk oracle meant to check `SEE()` - against) is wrapped in `#ifdef TEST_BROKEN` -- a define nothing - sets. `testsee.c`'s only *called* functions (`TestGetAttacks`, - `TestIsAttackedBB`) test attack generation, not SEE. Given SEE is - what `GetMoveSee`/extensions/QSearch pruning all trust (and is the - exact code reworked in commit c0ac3d7), this is the single - highest-value fix here. Get `TEST_BROKEN` -> live (`TEST`), fix - whatever bitrot surfaces, and confirm it's called from `main.c`. -2. **`TestEvalWithKnownPositions`** (`testeval.c`, ~20 hand-curated - "side X should be better here" positions from chess literature) is - defined but **never called from anywhere** -- not `main.c`, not - `command.c`. Wire it into the self-test sequence. -3. **`TestRecognExhaustiveKNKP`** (`testrecogn.c`, exhaustive KNKP - endgame-recognizer enumeration) is defined and prototyped in - `chess.h` but referenced only in a comment -- never invoked. Only - the smaller table-driven `TestRecogn` runs. Wire it in. -4. **The hash table has no correctness self-test at all.** - `testhash.c` contains `SanityCheckHashEntry` (an entry-shape - validator) and `AnalyzeFullHashTable` -- but the latter is a - post-search *diagnostic dump* called from `root.c`, not a test, and - it's never invoked from the `TEST` self-test sequence. See "New - test: hash correctness" below for what should replace this gap, not - just a wiring fix. -5. **`TestEvalWithSymmetry`** only runs a *single* random symmetric - position per invocation, and only when a human types `eval` at the - prompt in an `EVAL_DUMP` build (`command.c:2203`) -- not part of the - automated suite, never loops. See "New test: eval symmetry fuzz" - below. - -## New tests, by area - -### Search (thinnest area relative to its bug surface) - -- `TestSearch` (`testsearch.c`) runs 20 random positions at fixed - **depth 2** only, checking solely "returned move is legal" and - "didn't blow the time limit." - - **DONE -- hash-consistency test** (`TestSearchHashConsistency`, - testsearch.c): searches a position to depth D twice in a row with - a warm hash table (scores must match), and separately confirms a - depth-D2 search seeded by a shallower depth-D1 warm-up agrees with - a from-scratch depth-D2 search. Wired into the `TEST=1` self-test - sequence (main.c). - - **DONE (partially) -- QSearch sanity test** - (`TestQSearchSanity`, testsearch.c): checks search termination - (no `INVALID_SCORE`/garbage) and a whole-search node-count - ceiling. Caveat found and documented in the function's own header - comment: the self-test suite builds with plain `gmake TEST=1`, - which doesn't define `PERF_COUNTERS` -- every `sCounters` field - (including `u64QNodeCount`, the qsearch-only counter this was - originally meant to check) is wrapped in `chess.h`'s `INC()` macro, - a no-op without that define. Confirmed live (`sd 3; go` on a - `TEST`-only binary reports "0 qnodes" regardless of real qsearch - activity). The test currently checks - `ctx->sCounters.tree.u64TotalNodeCount` instead (root.c's - unconditional per-node `++`, always live) -- a real but strictly - weaker signal, since it can't distinguish "qsearch exploded" from - "full-width search exploded." Building a `PERF_COUNTERS`-enabled - `TEST` profile (or adding an always-live qsearch-only counter) is - the natural follow-up to close this for real. The hand-auditable- - tree idea (`Trace()` every qsearch node on a small position) is - still worth doing but wasn't implemented this session. - - **NOT DONE -- determinism test** (same position/depth searched - twice, PV must match): explicitly descoped -- Scott's call is that - a stable-PV requirement makes the test break every time move - ordering changes for unrelated reasons, which is just noise, not a - real regression signal. A same-*score* (not same-PV) check is - already covered by the hash-consistency test above. - - **NOT DONE -- MP-vs-single-thread agreement test**: still open. - `split.c` has no dedicated correctness test. - - **DONE (as an EPD suite, not a C test) -- null-move/zugzwang - soundness**: `tests/zugzwang.epd` (5 positions, run via the new - `epd` command -- see below) checks actual best-move outcome - instead of an internal counter. An earlier attempt asserted - against `ctx->sCounters.extension.uZugzwang` (the verification- - search hit counter) using the 3 classic literature zugzwang - positions, but none of them structurally reach that code path on - this build: 2 have too little material for - `WeShouldTryNullmovePruning`'s own `> 2` non-pawn-count gate to - attempt null-move pruning at all, and the 3rd (despite landing in - the eligible window) gets resolved directly off this box's local - Syzygy tablebase before search ever touches null-move logic -- - confirmed live via the `TABLEBASE MOVE` trace line. Removed in - favor of the EPD suite, which doesn't care about internal - mechanism, only the final answer. See `testsearch.c`'s comment - where `TestNullMoveZugzwang` used to be for the full writeup. - -**New infrastructure that came out of chasing the above (search.c -touches none of this, but it's now how any future EPD-based search -test should be added):** - -- **`epd <file>` engine command** (`EpdCommand`, script.c) parses and - runs a raw EPD suite file directly -- no perl preprocessing step - first. Backed by `ParseEpdRecord` (fen.c), real per-record EPD - parsing (4 position fields, optional FEN-style halfmove/fullmove - counters, then `;`-separated `bm`/`am`/`id` opcodes). -- **Found and fixed while building this**: `tests/convert_epd.pl` (the - old perl-based EPD-to-script converter) had a real, silent- - corruption bug -- its `bm`/`am` regex used `[^b]+` as the filler - between the board and the opcode, which breaks on *every* - Black-to-move record (the side-to-move field is literally the - character `b`), and since `$fen`/`$id`/`$solution` were unscoped - globals rather than per-line locals, a failed match silently carried - the *previous* record's values forward instead of erroring. - Confirmed live: feeding it a 5-position batch with one Black-to-move - entry produced a byte-for-byte duplicate of the previous record for - that one line, no diagnostic at all. The script is left in place - (marked superseded in its own header) in case something still shells - out to it, but new suites should use `epd` instead. - Cross-checked the new parser against the old pipeline on - `ecm_ringers`: `script` (old `.ep_` format) and `epd` (new - `tests/ecm_ringers.epd`, produced by `ep_to_epd.pl`, see next item) - produced byte-identical node counts and solve stats. -- **`tests/ep_to_epd.pl`**: one-off reverse converter, turns the - existing `setboard`/`solution`/`avoid`/`id`/`go` script-format - suites back into real EPD now that `epd` can read it directly. Run - against every genuine tactical suite in `tests/` (not `bench.ep_` or - `trial.ep_`, which use `sd`/`st`/`easy`/`quit` and aren't EPD - underneath) -- produced `.epd` siblings for `arasan`, - `ecm_confident_quick`, `ecm_confident_sample2`, `ecm_confident`, - `ecm_hard_quick`, `ecm_hard`, `ecm_late`, `ecm_quick`, `ecm_ringers`, - `ecm-gcp`, `ecm-hard`, `ecm`, `ecm016_only`, `LCT2`, `mes400`, - `newiq`, `quick`, `swiss`, `wac`. `ecm.epd` came out to 879 records, - matching the `606/879`/`622/879` solve-count baselines already - quoted elsewhere in CLAUDE.md. - -### Eval - -- Automate `TestEvalWithSymmetry` into a real fuzz loop (N random - symmetric positions, assert `Eval() == 0` for all, as part of the - standard self-test run) instead of the current single-shot - manual-only invocation. -- **Adversarial/degenerate-position term-bounds test**: 0-pawn - positions, 9-queen positions, fully-open-board positions -- extend - `GenerateRandomLegalPosition` or hand-build cases, assert no eval - term produces an out-of-range score (reuse the existing - `EvalTrace`/`EVAL_DUMP` term-tracing machinery for this). -- Audit for any **incrementally-maintained eval state** (pawn hash, - king-safety cache) and add a from-scratch-recompute-vs-incremental - consistency check if any exists -- classic "works until the hash - collides" bug class. -- **Eval speed regression test**: no benchmark currently exists for - eval itself despite it being the named NPS bottleneck vs. Crafty - (see CLAUDE.md). Same interleaved-comparison shape as the existing - `TestGenerate*Speed` benchmarks, broken down by game phase. - -### Generate (best-covered area, but gaps remain) - -- Perft (`TestMoveGenerator`) only checks to **ply 4** even though - `uLeaves[]` already has reference values to ply 6/7 sitting unused -- - extend to a `--perft-deep` slow path, and add perft-**divide** - (per-root-move leaf breakdown) to localize failures to a specific - move instead of just tripping a panic. -- Add perft positions specifically chosen to stress **en passant, - underpromotion, and castling-through-check** individually (the - current 5 FENs give general coverage but weren't chosen for this). -- `TestGenerateAllMovesSpeed` and the per-piece-type speed tests are - **pure microbenchmarks with no correctness assertion** -- they print - cycles and never diff the move *sets* produced by the mailbox vs. - bitboard generators being compared. Add a sorted-move-list-equality - assertion alongside each (both generators already run back-to-back - in the same loop, so this is nearly free) -- otherwise a bitboard - generator silently missing a move class just prints a suspiciously - fast number and passes. -- `GetMoveSee`'s new non-pawn-capture SEE path (c0ac3d7) has no - dedicated test yet -- covered by the SEE oracle fix above once live. - -### Move/Position plumbing - -- Reasonably solid already (`TestMakeUnmakeMove`, `TestExposesCheck`, - `TestIsAttacked`). Add a **long-game fuzz test**: play N random games - of ~200 plies each (not just 1-ply-at-a-time exercises), asserting - `VerifyPositionConsistency` and `ComputeSig`-from-scratch agreement - with the incrementally-maintained signature at every ply. Confirm - `VerifyPositionConsistency` is actually compiled live in the - `TEST`/`DEBUG` build, not just present in source. - -### Hash (currently zero correctness coverage) - -New test needed, not just a wiring fix: -- Store/probe roundtrip correctness. -- Replacement-scheme behavior under forced collisions (fill table past - capacity with distinct signatures, confirm replacement policy - behaves as designed, not just "doesn't crash"). -- `ADJUST_MATE_IN_N` ply-distance adjustment on `HASH_FLAG_EXACT` - entries specifically -- the exact mechanism CLAUDE.md flags as - audited-by-hand but not test-covered after this session's `ASSERT` - removals in `QSearch()`. - -### Recognizer - -- Wire in `TestRecognExhaustiveKNKP` (see dead-infra list above). - Consider the same exhaustive-enumeration treatment for any other - small recognized endgame class in `recogn.c` (KPK, KRK) if one - exists without a test. - -## Perf testing - -Current coverage is narrow: five isolated per-piece-type -mailbox-vs-bitboard microbenchmarks (`TestGenerate*Speed`), nothing for -search, eval, or hash. - -- **Eval NPS/cycles-per-call benchmark**, same interleaved-comparison - shape as the existing generate benchmarks, by game phase -- eval is - the named bottleneck (CLAUDE.md's 1.5M vs 7.5M nps gap vs. Crafty) - and has no dedicated speed test today. -- **Whole-search NPS benchmark**: `bench.c` already exists for this -- - confirm whether it's part of the automated `TEST` self-test pass or - purely a manual `bench` command; if manual-only, wire a - fixed-node-budget version into the automated sequence so a - regression surfaces without someone remembering to run it by hand. -- **Hash-probe-under-load benchmark**: cycles/probe at low occupancy - vs. near-full table -- replacement-scheme cost can regress silently - today with nothing to measure it. -- **Stored baselines with pass/fail thresholds**, not print-only - output: today every perf test prints cycles/call and leaves - interpretation to a human reading the log. Store a last-known-good - baseline (same idea as `head_reference/` for solve-counts) and fail - loud if a call site regresses past some tolerance. - -## Suggested priority order - -1. Un-bitrot the three dead tests (SEE oracle, - `TestEvalWithKnownPositions`, `TestRecognExhaustiveKNKP`) -- - near-zero cost, immediate coverage recovery. -2. Hash-table correctness test (store/probe/replace/mate-adjust) -- - currently the single biggest correctness blind spot. -3. Search determinism + MP-vs-single-thread agreement test -- targets - the exact class of bug most likely to hide in threading/hash/ - extension interactions that per-function unit tests can't see. -4. Extend perft to deeper plies + divide output; add move-set-equality - assertions to the existing generate speed benchmarks (turns - dead-weight microbenchmarks into correctness tests for free). -5. Eval: automate the symmetry fuzz test (loop N, not 1) and add - adversarial/degenerate-position term-bounds checks. -6. Round out perf testing per the list above, with stored baselines - and pass/fail thresholds instead of print-only output. - -## "Fuzz" testing - -**Implemented: `fuzz_test.sh [binary] [depth] [batch_size] [cpus]`** -(defaults: `./_typhoon`, `sd 6`, 200 positions/batch, `--cpus 2`). Loops -forever: each iteration draws a fresh random batch of `setboard`/`go` -records from the full 400,000-line / 200k-position -`tests/cross_sectional_200k.ep_` pool, runs them through a `DEBUG=1` -build under gdb, and scans the gdb output for a crash/assert signature -before starting the next batch. Stops (exit 1) and prints the offending -batch + engine log paths as soon as one is found; otherwise runs -indefinitely (`^C` to stop). Logs land under `/usr/local/tmp/typhoon/fuzz/` -(one sampled `.ep_` batch + one `.gdb.txt` + one engine `--logfile` per -iteration). - -**Two build/invocation bugs found and fixed getting this working, neither -a sandbox limitation despite first appearances:** - -1. **The DEBUG binary must be built with `MP=1` as well as `DEBUG=1`** - (`gmake clean; gmake DEBUG=1 MP=1`) -- `--cpus` is compiled in only - under `#ifdef MP` (`main.c:311`). A `DEBUG=1`-only build silently - treats `--cpus` and its numeric argument as two separate unrecognized - arguments (`main.c:400`'s "Error (unknown argument)" path) and falls - back to the 1-cpu default with no hard failure -- easy to miss, and it - also means `split.c`'s parallel-search code never actually links in, - defeating the point of passing `--cpus 2` at all. -2. **`gdb -batch -ex "handle SIGTRAP nostop noprint pass" -ex run ...` - breaks thread control against this MP/pthread build specifically**, - producing `ptrace (PT_SUSPEND): Operation not permitted` followed by - `Child process unexpectedly missing: No child processes` -- looked - exactly like a sandboxed environment blocking ptrace outright (and was - initially misdiagnosed as that), but reproduced identically outside - any harness/sandbox, and disappeared entirely once that one `-ex` line - was dropped. `fuzz_test.sh` omits it; a plain `run`/`bt full`/ - `info registers` gdb batch invocation debugs the MP binary fine, - including catching a genuine `int3` breakpoint from a failed `ASSERT`. - -Not yet done: no crash has been found by a run so far, so the -crash-detection path (the `grep -qE` in `fuzz_test.sh`) is exercised by -its own logic reading correctly, not yet by a real hit. Worth periodically -running this for stretches (it's unbounded/`^C`-driven, unlike the other -gates) rather than treating it as a one-time pass/fail check like -`debug_smoke_test.sh`. See also: `./debug_smoke_test.sh` (fast, curated -sample, single pass/fail) for the pre-commit-sized version of the same -idea. |
