path: root/src/CLAUDE.md

# typhoon chess engine

A chess engine by Scott Gasch ([email protected]), originally started 2004.
Previously named "monsoon" -- any old references to that name mean this same
codebase. Speaks the xboard/WinBoard protocol. FreeBSD host, 64-bit.

## Building

`build.sh`'s OS detection (`expr $OSTYPE = "darwin"`) is broken in this shell
environment and falls through to a 32-bit `gcc`-based profile that doesn't
exist here (no `gcc`, and the resulting `.o` files are architecture-mismatched
against everything else). **Don't use build.sh as-is.** Build directly with:

```sh
gmake -j5 GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1
```

This uses `clang`/`clang++` (already the GNUmakefile default) and produces a
64-bit binary matching the historically-built one. After changing any header
touched by many translation units, `gmake clean` first to avoid 32/64-bit
object mismatches at link time.

## Running / xboard protocol basics

No command-line flags are required for interactive use; the binary drops into
an xboard-protocol REPL on stdin/stdout. Key commands used during development:

- `force` -- stop the engine from auto-playing; you control both sides' moves.
- `setboard <fen>` -- set the position.
- `eval` -- print `Static eval: <score>` (calls `Eval()` directly, no search).
- `evaldna dump` / `evaldna read <file>` / `evaldna write <file>` -- the
  eval-constant import/export system (`g_EvalDNA` in eval.c). `evaldna` alone
  (no subcommand) just prints usage -- you need `evaldna dump` explicitly.
- `sd <n>` / `st <n>` -- fixed depth vs. fixed time search limits.
- `go` -- search and play one move for whichever side is on move, then stop
  (when paired with `force` after). Repeated `go`/`force` cycles correctly
  drive a full self-play game -- the engine tracks its own position across
  calls, no need to feed the opponent's move back manually.
- `script <file>` -- run a list of commands from a file, used for test suites
  (see ECM below). Prints solved/unsolved stats at the end.
- `--dnafile <path>` (startup flag) or `evaldna read <path>` (runtime) load a
  DNA file produced by tuning; `--egtbpath <path>` or `set EGTBPath <path>`
  configure Syzygy tablebase lookup.

**Default EGTB path is hardcoded** in `main.c:297` to `/zscratch/egtb`, used
whenever `--egtbpath` isn't passed. That directory currently has full 5-man
WDL+DTZ Syzygy coverage but is **missing the WDL half of its 6-man set** (only
`.rtbz`/DTZ files exist for 6-man combos) -- `init_tb()` in
`fathom/tbprobe.c:752` requires the WDL file to exist before registering a
combination at all, so every 6-man entry is silently skipped and the engine
reports "up to 5 men" even though 6-man `.rtbz` files are sitting right there.
Not a config issue; the fix is downloading the missing 6-man `.rtbw` files.

## Testing: the ECM tactical suite

`tests/ecm.ep_` is a 3517-line / 881-problem EPD tactical test suite.
`test.sh <epd-file> <seconds-per-move>` runs it and diffs against
`lastrun.log` via `suite_diff.pl`. Two things to know:

1. **`suite_diff.pl`'s shebang (`#!/usr/bin/perl`) is wrong on this box** --
   perl lives at `/usr/local/bin/perl`. Invoke it as `perl ./suite_diff.pl
   <new.log> <lastrun.log>` rather than `./suite_diff.pl`.
2. **Prefer `sd <depth>` over `st <seconds>` for before/after eval
   comparisons.** A time-based limit lets identical positions reach different
   search depths across runs purely from incidental machine load (this
   session saw large timing swings from unrelated processes/scrubs) --
   `sd 10` (or similar) holds search effort constant so a solve-rate
   difference is actually attributable to the eval change being tested.

Current baseline (`lastrun.log`, `st 1`, unmodified eval constants):
**606/879 solved.** A first full DNA-tuning pass against 600k TWIC positions
raised this to **622/879** (see `eval_tune/`).

`Trace()` (used for almost all engine output, including ECM's per-problem
results) only `fflush()`s stdout, **not** the logfile -- `Log()`/`Bug()` do
flush the logfile every call, but `Trace()` doesn't. A `--logfile` file can
lag well behind actual progress while a run is in flight; it catches up fully
once the process exits. Don't trust a logfile's line count as a live
progress indicator for a running batch job.

### The three curated suites and the `head_reference` protocol

Day-to-day search-change evaluation uses three small, curated EPD files
under `tests/` rather than the full 881-problem `ecm.ep_` (too slow to
iterate on) or a single random sample (too noisy -- a handful of flips
either way swamps the signal at that size):

- **`tests/ecm_ringers.ep_`** (11 positions) -- clean discriminators
  hand-picked because a baseline engine solves them reliably and a
  regressed one reliably doesn't (or vice versa). Fastest signal, but
  small enough that +/-1 solve is not necessarily real.
- **`tests/ecm_confident_quick.ep_`** (90 positions) -- a larger,
  still-fast sample the engine is expected to do well on.
- **`tests/ecm_hard_quick.ep_`** (90 positions) -- a harder sample,
  more sensitive to search-quality changes (pruning/reduction/ordering)
  since these positions need real depth/precision to crack.

Run all three together, not just one -- a change can look great on one
and cost solves on another (see `lmr_testing/RESULTS.md` for several
examples this cut both ways).

**Always compare against a fixed, known-good reference binary+logs, one
variable at a time -- never against "the working tree as it happened to
be a few hours ago."** `head_reference/` (sibling of `src/`, not
`tests/`) holds exactly this: a binary built from a specific commit,
plus full logs for all three suites at both `sd 10` (fixed depth) and
`sn 5000000` (fixed node budget per position) in `head_reference/logs/`,
and a `README.md` recording the commit hash and confirmed live/dead
state of every search mechanism (LMR, EFP, etc.) at that point --
confirmed by reading the code, not assumed from memory of what a
diagnostic build's `if (FALSE)` happened to say. Rebuild this directory
(and update its README) whenever committing a change that becomes the
new comparison baseline, so the next investigation always has
something to check itself against before drawing conclusions --
this exact gap (assuming a disabled-in-the-working-tree mechanism was
also disabled at HEAD, when it wasn't) cost a full afternoon
re-deriving EFP's actual committed behavior mid-investigation.

Pick `sd` vs `sn` based on what the change is expected to affect: `sd`
(fixed depth) is the right choice when judging whether a pruning/
ordering change makes the tree smaller or larger for the same search
effort; `sn` (fixed node budget) is the right choice when judging how
many positions solve within a fixed cost. Don't run both for every
single candidate change -- pick whichever matches what's actually being
tested (the reference logs in `head_reference/` keep both, precisely so
a later investigation can choose either without re-running HEAD).

## Eval tuning pipeline (`eval_tune/`)

`tune_eval_dna.py` is a Texel-style tuner: drives `typhoon` as an
xboard-protocol subprocess, fits `g_EvalDNA`'s ~1870 raw constants (reduced to
~468 after collapsing cells that share a starting value -- see caveat below)
against TWIC game outcomes via sigmoid-squared-error + coordinate descent.
`dna_diff.py` renders a human-readable, per-array diff between two `.dna`
files (with 8x8 grids for the board-shaped location tables) so you can
actually see what a tuning run changed. Position pools are cached under
`position_cache/` keyed on source-file mtime+size+filter args, since a
500k-600k position PGN scan takes ~20 minutes.

**Two real bugs were found and fixed this session, both in `command.c`, not
in the Python tooling:**

1. **`EvalCommand` used to malloc + fully zero (`InitializeSearcherContext`,
   which memsets the whole ~75MB `SEARCHER_THREAD_CONTEXT` -- `rgPawnHash`
   and `rgEvalHash` are embedded arrays inside that struct, not
   separately-managed heap buffers) on every single `eval` call, then free
   it.** Fixed to keep one persistent context and use the cheap
   `ReInitializeSearcherContext` (just repositions + zeros counters) on
   subsequent calls -- ~1000x throughput improvement for scripted eval
   scoring.
2. **That persistent context's embedded eval hash was never invalidated when
   new DNA was loaded**, so after the fix above, a position scored once under
   any DNA would return that *same cached score forever*, silently ignoring
   every later `evaldna read`. Fixed via a global `g_uDnaGeneration` counter
   (bumped in `EvalDnaCommand` on successful load, checked in `EvalCommand`)
   that forces exactly one full re-init per DNA change, not per position.
   **If this regresses, tuning results will look like it "converged" almost
   immediately with implausibly little movement -- that's the symptom, not
   genuine convergence.** Verify by perturbing a few known-live parameters
   and confirming the reported error actually changes.

**Known limitation, not yet fixed:** the distinct-value grouping used to
shrink 1870 raw cells to ~468 tunable parameters ties together *any* cells
that start out numerically equal, for whatever reason -- true mirror
symmetry, an intentional-but-asymmetric design choice (e.g.
`KING_INITIAL_COUNTER_BY_LOCATION`'s rank-8 row `1,0,0,0,1,0,0,1` flags
a1/e1/h1 specifically, not a geometric mirror), or pure incidental coincidence
(e.g. `PAWN_CENTRALITY_BONUS`'s legitimate "no bonus" squares happening to
share the value `0` with unrelated off-board padding cells). The padding case
is harmless (those indices are never read during real eval) but the
coincidental-equality case is a real, unaddressed source of noise in the
parameter space, most visible on small/noisy training runs (sentinel `-1`
placeholder cells in `TRADE_PIECES` drifted to an arbitrary shared value on a
25k-position smoke test; they stayed put on the full 600k run).

**`Engine`/`EnginePool` in `tune_eval_dna.py` must be used as a context
manager (`with Engine(...) as e:`) or have `.quit()` called explicitly.**
Each spawned engine holds a kernel SysV semaphore
(`semget(IPC_PRIVATE, ...)` in `unix.c`) for its input-dispatch loop, which is
**only released by a graceful `quit`, not by the OS when the process is
killed**. `kern.ipc.semmni` (max semaphore sets system-wide) is only 50 on
this box; a session's worth of force-killed test engines can leak past that
limit, after which *every new engine* silently falls back to a 100ms-per-
command polling path (`_WaitUntilTheresInputToRead` in `input.c:63`) instead
of the fast semaphore wait -- a ~1000x slowdown that looks exactly like "the
machine is under heavy load" and cost significant debugging time before the
real cause (`ipcs -s`, then `ipcrm -s <id>` for each leaked one) was found.

## Automated eval-tuning loop (`eval_tune/cycle.sh`)

A later session built a full wash/rinse/repeat pipeline on top of the manual
tuning tools above, so a cycle goes baseline -> tune -> head-to-head gate ->
bake-in-if-better, repeatable indefinitely against a growing game pool.

- **`filter_pgn.py <in.pgn> <out.pgn> --min-elo 2400 --min-ply 40`** -- fast
  text-only pre-filter (no python-chess board replay, so it runs in ~100s
  over a multi-GB pool) that drops short/low-rated games before they ever
  reach `tune_eval_dna.py`'s slower, board-replay-based filtering. Confirmed
  most of a raw TWIC pool's mass is short draws/low-rated games: filtering
  the full merged `twic.pgn` (4,099,094 games) down to >=2400 Elo / >40 ply
  kept 642,268 games (556MB) -- that's `twic_filtered.pgn`, the pool actually
  used for tuning and match play now.
- **`match_play.py <engine> <baseline.dna> <candidate.dna> --pgn <pool> --games N --sd D`**
  -- head-to-head gate match, N games (games/2 unique openings, each played
  twice with colors swapped, so opening luck cancels), fixed search depth
  (not clock time, same sd-over-st reasoning as ECM below). Prints
  `CANDIDATE_SCORE=... LOWER95=...` -- **gate on `LOWER95 >= 0.5`, not the raw
  score.** Search is non-deterministic (MP=1 multithreaded), confirmed live:
  two engines loaded with *identical* DNA still split games unevenly at
  small N, so a bare score >=0.5 is not evidence of a real improvement, only
  a 95%-confidence lower bound that still clears break-even is.
  Three real bugs were found and fixed getting this to give sane numbers:
    1. **The opening book was still live**, so early moves in every game
       were book lookups that never touched eval at all -- defeats the
       purpose of a match meant to compare eval quality. Fixed by pointing
       `book name` at a nonexistent file each game; `root.c`'s
       `g_uBookProbeFailures >= 3` gate then disables book probing for the
       rest of that game after 3 cheap failed opens.
    2. **Engines were spawned with no `--hash`/`--cpus` at all**, so every
       game ran on `main.c`'s memset-zero default (a 65536-entry hash table)
       instead of the 256m used everywhere else in this project's own
       testing -- worse move ordering, much more re-search in complex
       positions. Fixed by passing `--hash 256m --cpus 1` explicitly (added
       an `extra_args` param to `tune_eval_dna.py`'s `Engine` class for
       this).
    3. **Every engine process also defaulted to a *shared* `typhoon.log`**
       in cwd (same memset-zero default). With `--workers N` running
       `2*N` concurrent engines, they were all racing on
       `BackupFile()`-then-`fopen(wb+)` at startup and `fflush()`ing that
       one file on every `Log()`/`Bug()` call -- real lock contention,
       observed live as unexplained match slowdown. Fixed with
       `--logfile -`, which `main.c` special-cases to skip file logging
       entirely (no rename race, no fopen, no fflush).
    4. Separately (not a match_play.py bug but cost real wall-clock time
       chasing it as one): `sample_openings()`'s original implementation did
       a sequential `chess.pgn.read_game()` scan from byte 0 of the pool
       file to pick even a couple of random openings -- minutes of wasted
       I/O/parsing on a 556MB pool. Rewritten to seek to random byte
       offsets and parse just the one game found there, cached the same way
       as `tune_eval_dna.py`'s `position_cache/` (keyed on pool identity +
       params) under `opening_cache/`.
  **If a match run seems to hang or take far longer than a lone `sd N`
  search should:** check for leftover engine processes from a previous
  killed/interrupted run first (`ps aux | grep typhoon`) before assuming a
  new bug -- a stray process from an earlier debug session silently eating a
  full CPU core was mistaken for a pipeline bug for a while before being
  found and killed.
- **`bake_dna.py <candidate.dna> --eval-c eval.c --out eval.c`** -- writes a
  winning candidate's values into `eval.c`'s array initializers *in place*,
  preserving all formatting/comments/8x8 grid layout (regex-replaces only
  the numeric literals found outside comments, in `g_EvalDNA[]` order, so a
  resulting `git diff` shows only the numbers that actually changed).
  Round-trip verified: dumping the currently-compiled-in DNA and baking it
  right back produces a byte-identical `eval.c`. **Positional, not
  name-matched** -- a `.dna` file only works against the exact `eval.c`
  revision it was tuned against; a line-count mismatch against the current
  `g_EvalDNA[]` aborts loudly rather than silently misassigning values.
- **`dna_trend.py cycle1.dna cycle2.dna ... [--cycles-dir eval_tune/cycles]`**
  -- across a chronological sequence of *kept* candidates, flags whether
  each raw DNA cell is trending (every step moves it the same direction --
  real signal) or flip-flopping (moves a lot, nets out near zero -- chasing
  sampling noise in that cycle's position batch). Worth running every few
  kept cycles; a consistently flip-flopping cell is a candidate to exclude
  from future tuning passes rather than let it keep eating budget on noise.
- **`cycle.sh [pgn] [n_games] [workers] [max_positions] [max_passes] [sd_depth]`**
  -- wires all of the above into one cycle: dump baseline -> tune -> gate
  match -> bake-in-and-rebuild if `LOWER95 >= 0.5`, else discard and leave
  `eval.c` untouched. **Never runs `git commit`** -- a win leaves `eval.c`
  modified-but-uncommitted for manual review, by design.
  `sd_depth` defaults to 10; a dry run at that depth was observed still
  running after 15+ minutes on one game before the bugs above were found --
  use a shallow depth (4-6) for pipeline-wiring smoke tests, not 10.
- **`evaldna write <file>` refuses to overwrite an existing file** (silent
  `Error writing dna file.` with no explanation of why) -- `WriteEvalDNA` in
  eval.c does `if (SystemDoesFileExist(...)) goto end;` rather than
  clobbering. Delete the target first when re-dumping a baseline.
- **Watch for apples-to-oranges ECM comparisons across depths/time
  controls.** A baseline solved-count recorded at `st 1` is not a fair
  comparison point against a candidate scored at `sd 20` -- rescoring the
  baseline at the *same* fixed depth as the candidate revealed a real gap
  (711 vs 707 at `sd 20`, both far below the apparent 606-vs-711 jump you'd
  get comparing against the old `st 1` number) that was noise-level, not the
  large improvement it first looked like. Always diff ECM numbers taken at
  the same `sd` depth.

## Environment notes

- Shared, multi-user FreeBSD box with genuinely variable load (other Claude
  Code sessions, a `bhyve` VM, periodic `zpool scrub` on both `zroot` and
  `zscratch`). Don't trust a single throughput measurement without a
  back-to-back comparison under the same conditions; this session saw the
  same benchmark vary by 10x+ run to run for reasons that turned out to be
  the semaphore leak above, not genuine machine contention.
- `/zscratch/egtb` and `/usr/home/scott/typhoon/pgn/twic.pgn` are the two
  large external resources this codebase's tooling depends on. `twic.pgn`
  was originally 554,135 games (up through TWIC issue 608, fetched by the
  long-dead `pgn/twic.pl`); a later session fetched all subsequent issues
  (920-1658) directly from theweekinchess.com (its WAF blocks bare `wget`,
  needs a real browser `User-Agent` + `Accept` header via `curl`) and merged
  them in, growing it to 4,099,094 games (3.5GB). Note issues 609-919 are
  still a gap (fetched range starts at 920) if completeness ever matters.
  `pgn/twic_filtered.pgn` (642,268 games, 556MB, >=2400 Elo / >40 ply, via
  `filter_pgn.py`) is the pool actually used for tuning/match-play now, not
  the raw file.