diff options
Diffstat (limited to 'src/CLAUDE.md')
| -rw-r--r-- | src/CLAUDE.md | 257 |
1 files changed, 257 insertions, 0 deletions
diff --git a/src/CLAUDE.md b/src/CLAUDE.md new file mode 100644 index 0000000..881b062 --- /dev/null +++ b/src/CLAUDE.md @@ -0,0 +1,257 @@ +# typhoon chess engine + +A chess engine by Scott Gasch ([email protected]), originally started 2004. +Previously named "monsoon" -- any old references to that name mean this same +codebase. Speaks the xboard/WinBoard protocol. FreeBSD host, 64-bit. + +## Building + +`build.sh`'s OS detection (`expr $OSTYPE = "darwin"`) is broken in this shell +environment and falls through to a 32-bit `gcc`-based profile that doesn't +exist here (no `gcc`, and the resulting `.o` files are architecture-mismatched +against everything else). **Don't use build.sh as-is.** Build directly with: + +```sh +gmake -j5 GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1 +``` + +This uses `clang`/`clang++` (already the GNUmakefile default) and produces a +64-bit binary matching the historically-built one. After changing any header +touched by many translation units, `gmake clean` first to avoid 32/64-bit +object mismatches at link time. + +## Running / xboard protocol basics + +No command-line flags are required for interactive use; the binary drops into +an xboard-protocol REPL on stdin/stdout. Key commands used during development: + +- `force` -- stop the engine from auto-playing; you control both sides' moves. +- `setboard <fen>` -- set the position. +- `eval` -- print `Static eval: <score>` (calls `Eval()` directly, no search). +- `evaldna dump` / `evaldna read <file>` / `evaldna write <file>` -- the + eval-constant import/export system (`g_EvalDNA` in eval.c). `evaldna` alone + (no subcommand) just prints usage -- you need `evaldna dump` explicitly. +- `sd <n>` / `st <n>` -- fixed depth vs. fixed time search limits. +- `go` -- search and play one move for whichever side is on move, then stop + (when paired with `force` after). Repeated `go`/`force` cycles correctly + drive a full self-play game -- the engine tracks its own position across + calls, no need to feed the opponent's move back manually. +- `script <file>` -- run a list of commands from a file, used for test suites + (see ECM below). Prints solved/unsolved stats at the end. +- `--dnafile <path>` (startup flag) or `evaldna read <path>` (runtime) load a + DNA file produced by tuning; `--egtbpath <path>` or `set EGTBPath <path>` + configure Syzygy tablebase lookup. + +**Default EGTB path is hardcoded** in `main.c:297` to `/zscratch/egtb`, used +whenever `--egtbpath` isn't passed. That directory currently has full 5-man +WDL+DTZ Syzygy coverage but is **missing the WDL half of its 6-man set** (only +`.rtbz`/DTZ files exist for 6-man combos) -- `init_tb()` in +`fathom/tbprobe.c:752` requires the WDL file to exist before registering a +combination at all, so every 6-man entry is silently skipped and the engine +reports "up to 5 men" even though 6-man `.rtbz` files are sitting right there. +Not a config issue; the fix is downloading the missing 6-man `.rtbw` files. + +## Testing: the ECM tactical suite + +`tests/ecm.ep_` is a 3517-line / 881-problem EPD tactical test suite. +`test.sh <epd-file> <seconds-per-move>` runs it and diffs against +`lastrun.log` via `suite_diff.pl`. Two things to know: + +1. **`suite_diff.pl`'s shebang (`#!/usr/bin/perl`) is wrong on this box** -- + perl lives at `/usr/local/bin/perl`. Invoke it as `perl ./suite_diff.pl + <new.log> <lastrun.log>` rather than `./suite_diff.pl`. +2. **Prefer `sd <depth>` over `st <seconds>` for before/after eval + comparisons.** A time-based limit lets identical positions reach different + search depths across runs purely from incidental machine load (this + session saw large timing swings from unrelated processes/scrubs) -- + `sd 10` (or similar) holds search effort constant so a solve-rate + difference is actually attributable to the eval change being tested. + +Current baseline (`lastrun.log`, `st 1`, unmodified eval constants): +**606/879 solved.** A first full DNA-tuning pass against 600k TWIC positions +raised this to **622/879** (see `eval_tune/`). + +`Trace()` (used for almost all engine output, including ECM's per-problem +results) only `fflush()`s stdout, **not** the logfile -- `Log()`/`Bug()` do +flush the logfile every call, but `Trace()` doesn't. A `--logfile` file can +lag well behind actual progress while a run is in flight; it catches up fully +once the process exits. Don't trust a logfile's line count as a live +progress indicator for a running batch job. + +## Eval tuning pipeline (`eval_tune/`) + +`tune_eval_dna.py` is a Texel-style tuner: drives `typhoon` as an +xboard-protocol subprocess, fits `g_EvalDNA`'s ~1870 raw constants (reduced to +~468 after collapsing cells that share a starting value -- see caveat below) +against TWIC game outcomes via sigmoid-squared-error + coordinate descent. +`dna_diff.py` renders a human-readable, per-array diff between two `.dna` +files (with 8x8 grids for the board-shaped location tables) so you can +actually see what a tuning run changed. Position pools are cached under +`position_cache/` keyed on source-file mtime+size+filter args, since a +500k-600k position PGN scan takes ~20 minutes. + +**Two real bugs were found and fixed this session, both in `command.c`, not +in the Python tooling:** + +1. **`EvalCommand` used to malloc + fully zero (`InitializeSearcherContext`, + which memsets the whole ~75MB `SEARCHER_THREAD_CONTEXT` -- `rgPawnHash` + and `rgEvalHash` are embedded arrays inside that struct, not + separately-managed heap buffers) on every single `eval` call, then free + it.** Fixed to keep one persistent context and use the cheap + `ReInitializeSearcherContext` (just repositions + zeros counters) on + subsequent calls -- ~1000x throughput improvement for scripted eval + scoring. +2. **That persistent context's embedded eval hash was never invalidated when + new DNA was loaded**, so after the fix above, a position scored once under + any DNA would return that *same cached score forever*, silently ignoring + every later `evaldna read`. Fixed via a global `g_uDnaGeneration` counter + (bumped in `EvalDnaCommand` on successful load, checked in `EvalCommand`) + that forces exactly one full re-init per DNA change, not per position. + **If this regresses, tuning results will look like it "converged" almost + immediately with implausibly little movement -- that's the symptom, not + genuine convergence.** Verify by perturbing a few known-live parameters + and confirming the reported error actually changes. + +**Known limitation, not yet fixed:** the distinct-value grouping used to +shrink 1870 raw cells to ~468 tunable parameters ties together *any* cells +that start out numerically equal, for whatever reason -- true mirror +symmetry, an intentional-but-asymmetric design choice (e.g. +`KING_INITIAL_COUNTER_BY_LOCATION`'s rank-8 row `1,0,0,0,1,0,0,1` flags +a1/e1/h1 specifically, not a geometric mirror), or pure incidental coincidence +(e.g. `PAWN_CENTRALITY_BONUS`'s legitimate "no bonus" squares happening to +share the value `0` with unrelated off-board padding cells). The padding case +is harmless (those indices are never read during real eval) but the +coincidental-equality case is a real, unaddressed source of noise in the +parameter space, most visible on small/noisy training runs (sentinel `-1` +placeholder cells in `TRADE_PIECES` drifted to an arbitrary shared value on a +25k-position smoke test; they stayed put on the full 600k run). + +**`Engine`/`EnginePool` in `tune_eval_dna.py` must be used as a context +manager (`with Engine(...) as e:`) or have `.quit()` called explicitly.** +Each spawned engine holds a kernel SysV semaphore +(`semget(IPC_PRIVATE, ...)` in `unix.c`) for its input-dispatch loop, which is +**only released by a graceful `quit`, not by the OS when the process is +killed**. `kern.ipc.semmni` (max semaphore sets system-wide) is only 50 on +this box; a session's worth of force-killed test engines can leak past that +limit, after which *every new engine* silently falls back to a 100ms-per- +command polling path (`_WaitUntilTheresInputToRead` in `input.c:63`) instead +of the fast semaphore wait -- a ~1000x slowdown that looks exactly like "the +machine is under heavy load" and cost significant debugging time before the +real cause (`ipcs -s`, then `ipcrm -s <id>` for each leaked one) was found. + +## Automated eval-tuning loop (`eval_tune/cycle.sh`) + +A later session built a full wash/rinse/repeat pipeline on top of the manual +tuning tools above, so a cycle goes baseline -> tune -> head-to-head gate -> +bake-in-if-better, repeatable indefinitely against a growing game pool. + +- **`filter_pgn.py <in.pgn> <out.pgn> --min-elo 2400 --min-ply 40`** -- fast + text-only pre-filter (no python-chess board replay, so it runs in ~100s + over a multi-GB pool) that drops short/low-rated games before they ever + reach `tune_eval_dna.py`'s slower, board-replay-based filtering. Confirmed + most of a raw TWIC pool's mass is short draws/low-rated games: filtering + the full merged `twic.pgn` (4,099,094 games) down to >=2400 Elo / >40 ply + kept 642,268 games (556MB) -- that's `twic_filtered.pgn`, the pool actually + used for tuning and match play now. +- **`match_play.py <engine> <baseline.dna> <candidate.dna> --pgn <pool> --games N --sd D`** + -- head-to-head gate match, N games (games/2 unique openings, each played + twice with colors swapped, so opening luck cancels), fixed search depth + (not clock time, same sd-over-st reasoning as ECM below). Prints + `CANDIDATE_SCORE=... LOWER95=...` -- **gate on `LOWER95 >= 0.5`, not the raw + score.** Search is non-deterministic (MP=1 multithreaded), confirmed live: + two engines loaded with *identical* DNA still split games unevenly at + small N, so a bare score >=0.5 is not evidence of a real improvement, only + a 95%-confidence lower bound that still clears break-even is. + Three real bugs were found and fixed getting this to give sane numbers: + 1. **The opening book was still live**, so early moves in every game + were book lookups that never touched eval at all -- defeats the + purpose of a match meant to compare eval quality. Fixed by pointing + `book name` at a nonexistent file each game; `root.c`'s + `g_uBookProbeFailures >= 3` gate then disables book probing for the + rest of that game after 3 cheap failed opens. + 2. **Engines were spawned with no `--hash`/`--cpus` at all**, so every + game ran on `main.c`'s memset-zero default (a 65536-entry hash table) + instead of the 256m used everywhere else in this project's own + testing -- worse move ordering, much more re-search in complex + positions. Fixed by passing `--hash 256m --cpus 1` explicitly (added + an `extra_args` param to `tune_eval_dna.py`'s `Engine` class for + this). + 3. **Every engine process also defaulted to a *shared* `typhoon.log`** + in cwd (same memset-zero default). With `--workers N` running + `2*N` concurrent engines, they were all racing on + `BackupFile()`-then-`fopen(wb+)` at startup and `fflush()`ing that + one file on every `Log()`/`Bug()` call -- real lock contention, + observed live as unexplained match slowdown. Fixed with + `--logfile -`, which `main.c` special-cases to skip file logging + entirely (no rename race, no fopen, no fflush). + 4. Separately (not a match_play.py bug but cost real wall-clock time + chasing it as one): `sample_openings()`'s original implementation did + a sequential `chess.pgn.read_game()` scan from byte 0 of the pool + file to pick even a couple of random openings -- minutes of wasted + I/O/parsing on a 556MB pool. Rewritten to seek to random byte + offsets and parse just the one game found there, cached the same way + as `tune_eval_dna.py`'s `position_cache/` (keyed on pool identity + + params) under `opening_cache/`. + **If a match run seems to hang or take far longer than a lone `sd N` + search should:** check for leftover engine processes from a previous + killed/interrupted run first (`ps aux | grep typhoon`) before assuming a + new bug -- a stray process from an earlier debug session silently eating a + full CPU core was mistaken for a pipeline bug for a while before being + found and killed. +- **`bake_dna.py <candidate.dna> --eval-c eval.c --out eval.c`** -- writes a + winning candidate's values into `eval.c`'s array initializers *in place*, + preserving all formatting/comments/8x8 grid layout (regex-replaces only + the numeric literals found outside comments, in `g_EvalDNA[]` order, so a + resulting `git diff` shows only the numbers that actually changed). + Round-trip verified: dumping the currently-compiled-in DNA and baking it + right back produces a byte-identical `eval.c`. **Positional, not + name-matched** -- a `.dna` file only works against the exact `eval.c` + revision it was tuned against; a line-count mismatch against the current + `g_EvalDNA[]` aborts loudly rather than silently misassigning values. +- **`dna_trend.py cycle1.dna cycle2.dna ... [--cycles-dir eval_tune/cycles]`** + -- across a chronological sequence of *kept* candidates, flags whether + each raw DNA cell is trending (every step moves it the same direction -- + real signal) or flip-flopping (moves a lot, nets out near zero -- chasing + sampling noise in that cycle's position batch). Worth running every few + kept cycles; a consistently flip-flopping cell is a candidate to exclude + from future tuning passes rather than let it keep eating budget on noise. +- **`cycle.sh [pgn] [n_games] [workers] [max_positions] [max_passes] [sd_depth]`** + -- wires all of the above into one cycle: dump baseline -> tune -> gate + match -> bake-in-and-rebuild if `LOWER95 >= 0.5`, else discard and leave + `eval.c` untouched. **Never runs `git commit`** -- a win leaves `eval.c` + modified-but-uncommitted for manual review, by design. + `sd_depth` defaults to 10; a dry run at that depth was observed still + running after 15+ minutes on one game before the bugs above were found -- + use a shallow depth (4-6) for pipeline-wiring smoke tests, not 10. +- **`evaldna write <file>` refuses to overwrite an existing file** (silent + `Error writing dna file.` with no explanation of why) -- `WriteEvalDNA` in + eval.c does `if (SystemDoesFileExist(...)) goto end;` rather than + clobbering. Delete the target first when re-dumping a baseline. +- **Watch for apples-to-oranges ECM comparisons across depths/time + controls.** A baseline solved-count recorded at `st 1` is not a fair + comparison point against a candidate scored at `sd 20` -- rescoring the + baseline at the *same* fixed depth as the candidate revealed a real gap + (711 vs 707 at `sd 20`, both far below the apparent 606-vs-711 jump you'd + get comparing against the old `st 1` number) that was noise-level, not the + large improvement it first looked like. Always diff ECM numbers taken at + the same `sd` depth. + +## Environment notes + +- Shared, multi-user FreeBSD box with genuinely variable load (other Claude + Code sessions, a `bhyve` VM, periodic `zpool scrub` on both `zroot` and + `zscratch`). Don't trust a single throughput measurement without a + back-to-back comparison under the same conditions; this session saw the + same benchmark vary by 10x+ run to run for reasons that turned out to be + the semaphore leak above, not genuine machine contention. +- `/zscratch/egtb` and `/usr/home/scott/typhoon/pgn/twic.pgn` are the two + large external resources this codebase's tooling depends on. `twic.pgn` + was originally 554,135 games (up through TWIC issue 608, fetched by the + long-dead `pgn/twic.pl`); a later session fetched all subsequent issues + (920-1658) directly from theweekinchess.com (its WAF blocks bare `wget`, + needs a real browser `User-Agent` + `Accept` header via `curl`) and merged + them in, growing it to 4,099,094 games (3.5GB). Note issues 609-919 are + still a gap (fetched range starts at 920) if completeness ever matters. + `pgn/twic_filtered.pgn` (642,268 games, 556MB, >=2400 Elo / >40 ply, via + `filter_pgn.py`) is the pool actually used for tuning/match-play now, not + the raw file. |