Started doing texel eval tuning.

1 files changed, 257 insertions, 0 deletions

diff --git a/src/CLAUDE.md b/src/CLAUDE.md
new file mode 100644
index 0000000..881b062
--- /dev/null
+++ b/src/CLAUDE.md
@@ -0,0 +1,257 @@
+# typhoon chess engine
+
+A chess engine by Scott Gasch ([email protected]), originally started 2004.
+Previously named "monsoon" -- any old references to that name mean this same
+codebase. Speaks the xboard/WinBoard protocol. FreeBSD host, 64-bit.
+
+## Building
+
+`build.sh`'s OS detection (`expr $OSTYPE = "darwin"`) is broken in this shell
+environment and falls through to a 32-bit `gcc`-based profile that doesn't
+exist here (no `gcc`, and the resulting `.o` files are architecture-mismatched
+against everything else). **Don't use build.sh as-is.** Build directly with:
+
+```sh
+gmake -j5 GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1
+```
+
+This uses `clang`/`clang++` (already the GNUmakefile default) and produces a
+64-bit binary matching the historically-built one. After changing any header
+touched by many translation units, `gmake clean` first to avoid 32/64-bit
+object mismatches at link time.
+
+## Running / xboard protocol basics
+
+No command-line flags are required for interactive use; the binary drops into
+an xboard-protocol REPL on stdin/stdout. Key commands used during development:
+
+- `force` -- stop the engine from auto-playing; you control both sides' moves.
+- `setboard <fen>` -- set the position.
+- `eval` -- print `Static eval: <score>` (calls `Eval()` directly, no search).
+- `evaldna dump` / `evaldna read <file>` / `evaldna write <file>` -- the
+  eval-constant import/export system (`g_EvalDNA` in eval.c). `evaldna` alone
+  (no subcommand) just prints usage -- you need `evaldna dump` explicitly.
+- `sd <n>` / `st <n>` -- fixed depth vs. fixed time search limits.
+- `go` -- search and play one move for whichever side is on move, then stop
+  (when paired with `force` after). Repeated `go`/`force` cycles correctly
+  drive a full self-play game -- the engine tracks its own position across
+  calls, no need to feed the opponent's move back manually.
+- `script <file>` -- run a list of commands from a file, used for test suites
+  (see ECM below). Prints solved/unsolved stats at the end.
+- `--dnafile <path>` (startup flag) or `evaldna read <path>` (runtime) load a
+  DNA file produced by tuning; `--egtbpath <path>` or `set EGTBPath <path>`
+  configure Syzygy tablebase lookup.
+
+**Default EGTB path is hardcoded** in `main.c:297` to `/zscratch/egtb`, used
+whenever `--egtbpath` isn't passed. That directory currently has full 5-man
+WDL+DTZ Syzygy coverage but is **missing the WDL half of its 6-man set** (only
+`.rtbz`/DTZ files exist for 6-man combos) -- `init_tb()` in
+`fathom/tbprobe.c:752` requires the WDL file to exist before registering a
+combination at all, so every 6-man entry is silently skipped and the engine
+reports "up to 5 men" even though 6-man `.rtbz` files are sitting right there.
+Not a config issue; the fix is downloading the missing 6-man `.rtbw` files.
+
+## Testing: the ECM tactical suite
+
+`tests/ecm.ep_` is a 3517-line / 881-problem EPD tactical test suite.
+`test.sh <epd-file> <seconds-per-move>` runs it and diffs against
+`lastrun.log` via `suite_diff.pl`. Two things to know:
+
+1. **`suite_diff.pl`'s shebang (`#!/usr/bin/perl`) is wrong on this box** --
+   perl lives at `/usr/local/bin/perl`. Invoke it as `perl ./suite_diff.pl
+   <new.log> <lastrun.log>` rather than `./suite_diff.pl`.
+2. **Prefer `sd <depth>` over `st <seconds>` for before/after eval
+   comparisons.** A time-based limit lets identical positions reach different
+   search depths across runs purely from incidental machine load (this
+   session saw large timing swings from unrelated processes/scrubs) --
+   `sd 10` (or similar) holds search effort constant so a solve-rate
+   difference is actually attributable to the eval change being tested.
+
+Current baseline (`lastrun.log`, `st 1`, unmodified eval constants):
+**606/879 solved.** A first full DNA-tuning pass against 600k TWIC positions
+raised this to **622/879** (see `eval_tune/`).
+
+`Trace()` (used for almost all engine output, including ECM's per-problem
+results) only `fflush()`s stdout, **not** the logfile -- `Log()`/`Bug()` do
+flush the logfile every call, but `Trace()` doesn't. A `--logfile` file can
+lag well behind actual progress while a run is in flight; it catches up fully
+once the process exits. Don't trust a logfile's line count as a live
+progress indicator for a running batch job.
+
+## Eval tuning pipeline (`eval_tune/`)
+
+`tune_eval_dna.py` is a Texel-style tuner: drives `typhoon` as an
+xboard-protocol subprocess, fits `g_EvalDNA`'s ~1870 raw constants (reduced to
+~468 after collapsing cells that share a starting value -- see caveat below)
+against TWIC game outcomes via sigmoid-squared-error + coordinate descent.
+`dna_diff.py` renders a human-readable, per-array diff between two `.dna`
+files (with 8x8 grids for the board-shaped location tables) so you can
+actually see what a tuning run changed. Position pools are cached under
+`position_cache/` keyed on source-file mtime+size+filter args, since a
+500k-600k position PGN scan takes ~20 minutes.
+
+**Two real bugs were found and fixed this session, both in `command.c`, not
+in the Python tooling:**
+
+1. **`EvalCommand` used to malloc + fully zero (`InitializeSearcherContext`,
+   which memsets the whole ~75MB `SEARCHER_THREAD_CONTEXT` -- `rgPawnHash`
+   and `rgEvalHash` are embedded arrays inside that struct, not
+   separately-managed heap buffers) on every single `eval` call, then free
+   it.** Fixed to keep one persistent context and use the cheap
+   `ReInitializeSearcherContext` (just repositions + zeros counters) on
+   subsequent calls -- ~1000x throughput improvement for scripted eval
+   scoring.
+2. **That persistent context's embedded eval hash was never invalidated when
+   new DNA was loaded**, so after the fix above, a position scored once under
+   any DNA would return that *same cached score forever*, silently ignoring
+   every later `evaldna read`. Fixed via a global `g_uDnaGeneration` counter
+   (bumped in `EvalDnaCommand` on successful load, checked in `EvalCommand`)
+   that forces exactly one full re-init per DNA change, not per position.
+   **If this regresses, tuning results will look like it "converged" almost
+   immediately with implausibly little movement -- that's the symptom, not
+   genuine convergence.** Verify by perturbing a few known-live parameters
+   and confirming the reported error actually changes.
+
+**Known limitation, not yet fixed:** the distinct-value grouping used to
+shrink 1870 raw cells to ~468 tunable parameters ties together *any* cells
+that start out numerically equal, for whatever reason -- true mirror
+symmetry, an intentional-but-asymmetric design choice (e.g.
+`KING_INITIAL_COUNTER_BY_LOCATION`'s rank-8 row `1,0,0,0,1,0,0,1` flags
+a1/e1/h1 specifically, not a geometric mirror), or pure incidental coincidence
+(e.g. `PAWN_CENTRALITY_BONUS`'s legitimate "no bonus" squares happening to
+share the value `0` with unrelated off-board padding cells). The padding case
+is harmless (those indices are never read during real eval) but the
+coincidental-equality case is a real, unaddressed source of noise in the
+parameter space, most visible on small/noisy training runs (sentinel `-1`
+placeholder cells in `TRADE_PIECES` drifted to an arbitrary shared value on a
+25k-position smoke test; they stayed put on the full 600k run).
+
+**`Engine`/`EnginePool` in `tune_eval_dna.py` must be used as a context
+manager (`with Engine(...) as e:`) or have `.quit()` called explicitly.**
+Each spawned engine holds a kernel SysV semaphore
+(`semget(IPC_PRIVATE, ...)` in `unix.c`) for its input-dispatch loop, which is
+**only released by a graceful `quit`, not by the OS when the process is
+killed**. `kern.ipc.semmni` (max semaphore sets system-wide) is only 50 on
+this box; a session's worth of force-killed test engines can leak past that
+limit, after which *every new engine* silently falls back to a 100ms-per-
+command polling path (`_WaitUntilTheresInputToRead` in `input.c:63`) instead
+of the fast semaphore wait -- a ~1000x slowdown that looks exactly like "the
+machine is under heavy load" and cost significant debugging time before the
+real cause (`ipcs -s`, then `ipcrm -s <id>` for each leaked one) was found.
+
+## Automated eval-tuning loop (`eval_tune/cycle.sh`)
+
+A later session built a full wash/rinse/repeat pipeline on top of the manual
+tuning tools above, so a cycle goes baseline -> tune -> head-to-head gate ->
+bake-in-if-better, repeatable indefinitely against a growing game pool.
+
+- **`filter_pgn.py <in.pgn> <out.pgn> --min-elo 2400 --min-ply 40`** -- fast
+  text-only pre-filter (no python-chess board replay, so it runs in ~100s
+  over a multi-GB pool) that drops short/low-rated games before they ever
+  reach `tune_eval_dna.py`'s slower, board-replay-based filtering. Confirmed
+  most of a raw TWIC pool's mass is short draws/low-rated games: filtering
+  the full merged `twic.pgn` (4,099,094 games) down to >=2400 Elo / >40 ply
+  kept 642,268 games (556MB) -- that's `twic_filtered.pgn`, the pool actually
+  used for tuning and match play now.
+- **`match_play.py <engine> <baseline.dna> <candidate.dna> --pgn <pool> --games N --sd D`**
+  -- head-to-head gate match, N games (games/2 unique openings, each played
+  twice with colors swapped, so opening luck cancels), fixed search depth
+  (not clock time, same sd-over-st reasoning as ECM below). Prints
+  `CANDIDATE_SCORE=... LOWER95=...` -- **gate on `LOWER95 >= 0.5`, not the raw
+  score.** Search is non-deterministic (MP=1 multithreaded), confirmed live:
+  two engines loaded with *identical* DNA still split games unevenly at
+  small N, so a bare score >=0.5 is not evidence of a real improvement, only
+  a 95%-confidence lower bound that still clears break-even is.
+  Three real bugs were found and fixed getting this to give sane numbers:
+    1. **The opening book was still live**, so early moves in every game
+       were book lookups that never touched eval at all -- defeats the
+       purpose of a match meant to compare eval quality. Fixed by pointing
+       `book name` at a nonexistent file each game; `root.c`'s
+       `g_uBookProbeFailures >= 3` gate then disables book probing for the
+       rest of that game after 3 cheap failed opens.
+    2. **Engines were spawned with no `--hash`/`--cpus` at all**, so every
+       game ran on `main.c`'s memset-zero default (a 65536-entry hash table)
+       instead of the 256m used everywhere else in this project's own
+       testing -- worse move ordering, much more re-search in complex
+       positions. Fixed by passing `--hash 256m --cpus 1` explicitly (added
+       an `extra_args` param to `tune_eval_dna.py`'s `Engine` class for
+       this).
+    3. **Every engine process also defaulted to a *shared* `typhoon.log`**
+       in cwd (same memset-zero default). With `--workers N` running
+       `2*N` concurrent engines, they were all racing on
+       `BackupFile()`-then-`fopen(wb+)` at startup and `fflush()`ing that
+       one file on every `Log()`/`Bug()` call -- real lock contention,
+       observed live as unexplained match slowdown. Fixed with
+       `--logfile -`, which `main.c` special-cases to skip file logging
+       entirely (no rename race, no fopen, no fflush).
+    4. Separately (not a match_play.py bug but cost real wall-clock time
+       chasing it as one): `sample_openings()`'s original implementation did
+       a sequential `chess.pgn.read_game()` scan from byte 0 of the pool
+       file to pick even a couple of random openings -- minutes of wasted
+       I/O/parsing on a 556MB pool. Rewritten to seek to random byte
+       offsets and parse just the one game found there, cached the same way
+       as `tune_eval_dna.py`'s `position_cache/` (keyed on pool identity +
+       params) under `opening_cache/`.
+  **If a match run seems to hang or take far longer than a lone `sd N`
+  search should:** check for leftover engine processes from a previous
+  killed/interrupted run first (`ps aux | grep typhoon`) before assuming a
+  new bug -- a stray process from an earlier debug session silently eating a
+  full CPU core was mistaken for a pipeline bug for a while before being
+  found and killed.
+- **`bake_dna.py <candidate.dna> --eval-c eval.c --out eval.c`** -- writes a
+  winning candidate's values into `eval.c`'s array initializers *in place*,
+  preserving all formatting/comments/8x8 grid layout (regex-replaces only
+  the numeric literals found outside comments, in `g_EvalDNA[]` order, so a
+  resulting `git diff` shows only the numbers that actually changed).
+  Round-trip verified: dumping the currently-compiled-in DNA and baking it
+  right back produces a byte-identical `eval.c`. **Positional, not
+  name-matched** -- a `.dna` file only works against the exact `eval.c`
+  revision it was tuned against; a line-count mismatch against the current
+  `g_EvalDNA[]` aborts loudly rather than silently misassigning values.
+- **`dna_trend.py cycle1.dna cycle2.dna ... [--cycles-dir eval_tune/cycles]`**
+  -- across a chronological sequence of *kept* candidates, flags whether
+  each raw DNA cell is trending (every step moves it the same direction --
+  real signal) or flip-flopping (moves a lot, nets out near zero -- chasing
+  sampling noise in that cycle's position batch). Worth running every few
+  kept cycles; a consistently flip-flopping cell is a candidate to exclude
+  from future tuning passes rather than let it keep eating budget on noise.
+- **`cycle.sh [pgn] [n_games] [workers] [max_positions] [max_passes] [sd_depth]`**
+  -- wires all of the above into one cycle: dump baseline -> tune -> gate
+  match -> bake-in-and-rebuild if `LOWER95 >= 0.5`, else discard and leave
+  `eval.c` untouched. **Never runs `git commit`** -- a win leaves `eval.c`
+  modified-but-uncommitted for manual review, by design.
+  `sd_depth` defaults to 10; a dry run at that depth was observed still
+  running after 15+ minutes on one game before the bugs above were found --
+  use a shallow depth (4-6) for pipeline-wiring smoke tests, not 10.
+- **`evaldna write <file>` refuses to overwrite an existing file** (silent
+  `Error writing dna file.` with no explanation of why) -- `WriteEvalDNA` in
+  eval.c does `if (SystemDoesFileExist(...)) goto end;` rather than
+  clobbering. Delete the target first when re-dumping a baseline.
+- **Watch for apples-to-oranges ECM comparisons across depths/time
+  controls.** A baseline solved-count recorded at `st 1` is not a fair
+  comparison point against a candidate scored at `sd 20` -- rescoring the
+  baseline at the *same* fixed depth as the candidate revealed a real gap
+  (711 vs 707 at `sd 20`, both far below the apparent 606-vs-711 jump you'd
+  get comparing against the old `st 1` number) that was noise-level, not the
+  large improvement it first looked like. Always diff ECM numbers taken at
+  the same `sd` depth.
+
+## Environment notes
+
+- Shared, multi-user FreeBSD box with genuinely variable load (other Claude
+  Code sessions, a `bhyve` VM, periodic `zpool scrub` on both `zroot` and
+  `zscratch`). Don't trust a single throughput measurement without a
+  back-to-back comparison under the same conditions; this session saw the
+  same benchmark vary by 10x+ run to run for reasons that turned out to be
+  the semaphore leak above, not genuine machine contention.
+- `/zscratch/egtb` and `/usr/home/scott/typhoon/pgn/twic.pgn` are the two
+  large external resources this codebase's tooling depends on. `twic.pgn`
+  was originally 554,135 games (up through TWIC issue 608, fetched by the
+  long-dead `pgn/twic.pl`); a later session fetched all subsequent issues
+  (920-1658) directly from theweekinchess.com (its WAF blocks bare `wget`,
+  needs a real browser `User-Agent` + `Accept` header via `curl`) and merged
+  them in, growing it to 4,099,094 games (3.5GB). Note issues 609-919 are
+  still a gap (fetched range starts at 920) if completeness ever matters.
+  `pgn/twic_filtered.pgn` (642,268 games, 556MB, >=2400 Elo / >40 ply, via
+  `filter_pgn.py`) is the pool actually used for tuning/match-play now, not
+  the raw file.