Add king-safety mate-mining regression pipeline; retune file-storm/

queen-proximity king-safety weights and add a back-flight bonus from it.

Mines real human-game checkmates and matched sound-draw controls
(eval_tune/mine_king_mates.py, shard_mine.py, extract_king_features.py,
match_controls.py, train_king_safety_model.py), then fits a logistic
regression against _EvalKing's raw per-defect breakdown (engine-side
support: chess.h's KING_DEFECT_DUMP_INFO, eval.c's KING_DEFECT_DUMP
instrumentation, command.c's evalkingdump command, all gated behind the
KING_DEFECT_DUMP=1 build flag) to see which defects actually predict
getting mated. See eval_tune/king_safety_data/HANDOFF.md for the full
methodology and findings.

Two rounds of retuning landed from this data so far, both validated via
sd10 curated-suite comparison plus per-position Crafty corroboration
(tests/ECM.md) and gated self-play:

- KingFileDefects/KingStormingPawnDefects (_ComputeFileStormDefects)
  bumped ~1.75-2x relative to the rest of the ring-scan defects -- the
  regression's most dominant, most stable tier-1 predictors, previously
  weighted the same as every other hand-picked defect.
- KING_QUEEN_PROXIMITY_DANGER rescaled from -8/count to -32/count, and
  a new KING_BACK_FLIGHT_BONUS (+1/square) term added for empty,
  unattacked ring squares on the side of the king away from the
  attacker -- a real, non-collinear protective signal the current
  EmptyUnattackedSquare table didn't credit at all. Both calibrated by
  unstandardizing the regression's coefficients and anchoring against
  KING_MISSING_ONE_CASTLE_OPTION's already-live -23 value (same
  direct-EVAL_TERM unit family).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01P9EDZVcqu6E2bNRm4zDrTf

1 files changed, 254 insertions, 0 deletions

diff --git a/src/eval_tune/king_safety_data/HANDOFF.md b/src/eval_tune/king_safety_data/HANDOFF.md
new file mode 100644
index 0000000..b05e3d2
--- /dev/null
+++ b/src/eval_tune/king_safety_data/HANDOFF.md
@@ -0,0 +1,254 @@
+# King-safety defect regression: handoff
+
+## Goal
+
+`_EvalKing` (eval.c) sums a pile of hand-picked "defect" constants into one
+scalar (`uCounter`), then runs that through a single lookup table
+(`KING_SAFETY_BY_COUNTER`). The hand-picked weights assume every defect
+counts equally, which is suspect -- e.g. a queen touching an undefended
+ring square and a knight touching one add the exact same `+1`. This
+project mines real games to see which defects actually predict getting
+mated, so `eval.c`'s weights can be informed by data instead of guesswork.
+
+**Method**: find real middlegame checkmates in human games, look at the
+losing king's position N plies earlier, and compare its raw defect
+pattern against a matched "sound" control position (drawn games) with the
+same attacker material and development level. Fit a logistic regression;
+features with a real, stable coefficient are candidates for reweighting
+in `eval.c`. See `CLAUDE.md`'s "King-safety eval improvement project"
+section (search for it) for the full narrative this file summarizes.
+
+## Where everything is
+
+- **Final training data**: `remine_v4/king_features_matched_v4.csv` --
+  86,362 rows (43,181 matched positive/control pairs). **Use this one.**
+- `remine_v4/merged.csv` -- raw mined FENs before feature extraction
+  (172,735 rows: 43,181 positives + 129,554 control candidates).
+- `remine_v4/king_features_v4.csv` -- all 172,735 rows with full
+  `_EvalKing` defect breakdown, before matching trims it down to the
+  86,362 confound-controlled rows.
+- Pipeline scripts live in `eval_tune/` (sibling of this dir), not here:
+  `mine_king_mates.py`, `shard_mine.py`, `extract_king_features.py`,
+  `match_controls.py`, `train_king_safety_model.py`.
+- Engine side: `chess.h`'s `KING_DEFECT_DUMP_INFO` struct,
+  `_EvalKing`'s `#ifdef KING_DEFECT_DUMP` instrumentation in eval.c, and
+  command.c's `evalkingdump` command -- all gated behind the
+  `KING_DEFECT_DUMP=1` gmake build flag (NOT part of a normal build).
+  `../typhoon_kingdump` is a saved binary built with that flag; rebuild
+  it (`gmake clean` first -- flag changes don't trigger recompilation on
+  their own, see "Gotchas" below) if `eval.c`/`chess.h`/`command.c`
+  change.
+
+## Final result (from `king_features_matched_v4.csv`)
+
+Logistic regression, 80/20 train/test split, standardized coefficients.
+Test AUC 0.91 (plausible for "engineered danger features vs. proven-sound
+draws", not suspiciously perfect). Ranked by |coefficient|, most-to-least
+trustworthy tier first (stability = full-population vs. top-tercile-army-
+strength fits agreeing):
+
+**Tier 1 -- dominant, very stable**: `filestormopenfile` (+0.57 to +0.80),
+`filestormpawn` (+0.51 to +0.54). These are eval.c's `KingFileDefects[3]`
+and `KingStormingPawnDefects[8]` tables (open/half-open file near the
+king; how far an enemy pawn has advanced up that file), already
+independent tables -- lowest-risk place to start a real retune.
+
+**Tier 2 -- strong, stable**: `front_offboard`, `front_def`, `sq4_offboard`,
+`front_undef`, `front_atk_minor`, `front_atk_rook`, `misscastleshort`,
+`front_flight` (+0.33 to +0.45). "front" = the diagonal/vertical ring
+squares one rank away, in the direction each side attacks toward (see
+"Feature schema" below for the exact slot mapping -- do NOT assume this
+means "in front of the king" the way a human would first guess; see the
+correction note there). `sq3`/`sq4` are the two same-rank side squares,
+kept individually resolved.
+
+**Tier 3 -- moderate, less stable**: `back_offboard` (drifts the most of
+any surviving feature, treat with suspicion), `sq3_undef`,
+`attackerarmyscaler`, `queenproximity`, `back_atk_minor`, `front_atk_queen`.
+
+**Confirmed noise -- do not retune based on this data**: `minorsathome`
+(coefficient ~0.001, was previously the #1 "signal" before two sampling
+bugs got fixed -- see "Bugs found" below), `initloc`, `castled` (exactly
+0.0000, fully collinear with `misscastleshort`/`misscastlelong`).
+
+## Translating this into eval.c -- confidence and plan
+
+This is NOT a "throw away eval.c and use the model" situation. The plan
+is targeted retuning of existing constants/tables, informed by relative
+coefficient *ratios*, not literal log-odds-to-centipawn conversion:
+
+1. **`KingFileDefects`/`KingStormingPawnDefects` deserve a real weight
+   increase relative to everything else** -- they're the strongest
+   signal by a clear margin and already live in their own small,
+   isolated tables (`_ComputeFileStormDefects` in eval.c). Lowest risk,
+   highest expected payoff, start here.
+2. **`EmptyAttackedSquare`/`EmptyUnattackedSquare`'s front/back asymmetry
+   is directionally right but probably too weak a spread** -- front-
+   bucket features consistently outrank back-bucket ones here by a good
+   margin.
+3. **`misscastleshort`'s constant likely deserves a bump.**
+4. **Don't touch `KING_INITIAL_COUNTER_BY_LOCATION` or the undeveloped-
+   minors term** based on this data -- both came back as noise.
+5. Neither `KingFileDefects`/`KingStormingPawnDefects` nor the
+   ring-scan tables are currently `DNA_ARRAY`-registered (checked: only
+   `KING_SAFETY_BY_COUNTER` and `KING_QUEEN_PROXIMITY_DANGER` are under
+   king-safety). Retuning them means a manual edit (optionally promoting
+   them to file scope + registering as DNA arrays for future tuning),
+   not "rerun the tuner."
+6. **Absolute magnitude has to be calibrated the normal way** -- run any
+   change through `sd10` ECM suite comparison against `head_reference/`
+   and `match_play.py`'s gated head-to-head match, same as any other
+   hand-tuned eval constant. The regression tells you *shape*
+   (which defects matter more relative to each other), not the right
+   centipawn magnitude for a chess engine's scoring scale.
+7. `CountKingSafetyDefects` (the cheap lazy-eval proxy) stays untouched
+   until a real change lands and is validated -- agreed early in this
+   project, still the right call.
+
+## Feature schema (read this before extending the pipeline)
+
+`KingSafetyDeltas` in eval.c (`-17,-16,-15,-1,+1,+15,+16,+17`) defines 8
+ring slots, indices 0-7 in that order. **Correction made mid-project**:
+slots 3 and 4 (deltas -1/+1) are the two SAME-RANK side squares, not
+"front of the king" -- an earlier description in conversation got this
+wrong. The diagonal/vertical slots (0,1,2 and 5,6,7) are the ones with a
+real front/back asymmetry, matching `EmptyAttackedSquare`'s own existing
+color-dependent weighting (WHITE's front is slots 0/1/2, BLACK's is
+5/6/7 -- see `_EvalKing`'s comments and `KING_RING_FRONT`-style logic).
+
+`extract_king_features.py` dumps full per-slot resolution (all 8 slots x
+4 piece types x several state flags) since the engine-side cost of doing
+so is free once you're calling `evalkingdump` anyway. `train_king_safety_model.py`
+is where slots 0/1/2/5/6/7 get bucketed into "front"/"back" sums
+(color-dependent per row's `mated_color_or_sampled_color`), while slots 3/4
+stay individually resolved -- this bucketing was added in this session
+after the diagonal per-slot coefficients turned out to be the noisiest,
+most sign-flipping entries in every full-vs-subset drift comparison.
+`filestorm` is deliberately NOT used as a training feature -- its two
+independent sub-components (`filestormopenfile`, `filestormpawn`) are
+used instead, since they map onto two already-separate eval.c tables and
+there's no reason to make the regression re-derive their relative
+importance from one pre-summed number.
+
+## Bugs found and fixed this session (read before trusting old data)
+
+If you find any `.csv` file in this project OTHER than
+`remine_v4/*_v4.csv` and its merged.csv, IT PREDATES ONE OR MORE OF THESE
+FIXES AND SHOULD NOT BE USED. (Older files were deleted during cleanup,
+but if this pipeline gets rerun and produces intermediate files again,
+this is the checklist for whether they're trustworthy.)
+
+1. **Bare-king endgame mates leaking into "middlegame" positives.** Only
+   checking the attacker's material (`>= 800cp`, roughly rook+bishop) let
+   trivial K+Q-vs-lone-K technique mates through. Fixed by also requiring
+   `total_non_pawn_piece_count(board) >= 8` (mirrors `_EvalKing`'s own
+   `pos->uNonPawnCount[WHITE][0] + pos->uNonPawnCount[BLACK][0] < 8`
+   endgame boundary exactly) -- see `is_middlegame()` in
+   `mine_king_mates.py`.
+2. **`min_ply` too low (10, i.e. move 5) for control sampling.** A large
+   fraction of the pool's draws are short draws-by-agreement close to
+   `filter_pgn.py`'s own 40-ply floor. Combined with a large end-guard
+   (staying clear of the game's end), the only valid sampling window on a
+   short draw was the early opening -- where minors are legitimately
+   still undeveloped for completely ordinary reasons. This alone caused
+   `minorsathome` to look like a massive, spurious "signal" (controls
+   showed `minorsathome>=2` in ~85% of samples). Verified the engine's
+   own computation was correct on a real developed-middlegame FEN --
+   100% a sampling-window bug, not an eval bug. Fixed: `min_ply` raised
+   to 20 (move 10), and a game too short to satisfy both `min_ply` and
+   the end-guard is now skipped entirely rather than force-sampled early.
+3. **Control-sampling ply selection secretly always picked the earliest
+   plies in a game's valid window.** `rng.shuffle(candidates)` followed
+   by `for idx in sorted(candidates)` completely undid the shuffle.
+   Fixed in two steps (the first fix wasn't sufficient on its own --
+   see the code comment in `mine_controls()` for why): randomly
+   subsample which plies to try, replay ascending through all of them
+   ONCE (required -- boards can only be walked forward), collect every
+   result that passes filters into a list, THEN randomly choose which
+   results to actually keep via `rng.sample` on that list. Verified:
+   sampled-ply-as-fraction-of-game-length now has a median around 0.46
+   with real tail into 60%+, instead of clustering at 20-30%.
+4. **Matching-radius bug treated two very differently-scaled confound
+   columns with one shared distance metric.** `attackerarmyscaler`
+   spans 0-31 (`ASSERT`-confirmed hard clamp ceiling), `minorsathome`
+   spans 0-4. A single Chebyshev radius meant a "tight" radius-2 match
+   still let `minorsathome` drift by up to 2 -- literally half its
+   entire range. This is why `minorsathome` was STILL the #1 "signal"
+   even after an initial matching attempt. Fixed: `match_controls.py`
+   now takes `--exact-cols` (must match exactly, no radius search --
+   `minorsathome` defaults here) and `--radius-cols` (allowed to expand
+   outward, `attackerarmyscaler` defaults here) as separate concepts.
+5. **KDTree-based nearest-neighbor matching matched almost nothing** (308
+   of 43,181 positives) because the match columns are low-cardinality
+   integers -- huge exact-value clusters mean a KDTree's "k nearest" are
+   massively tied, so the first few positives processed in a cluster
+   exhaust the shared candidate set and everyone else in that cluster
+   fails to match. Replaced with exact-bucket + expanding-integer-radius
+   matching (see `match_controls.py`), which has no such collision
+   problem.
+6. **`_EvalKing`'s per-square ring scan implicitly assumed all 8 slots
+   were geometrically equivalent for feature purposes**; slots 3/4
+   (side squares) vs. the diagonal slots turned out to behave
+   differently enough that per-slot resolution on the diagonals was
+   pure noise at the sample sizes available -- see "Feature schema"
+   above.
+7. **Board-diagram spam.** `setboard` prints a full ASCII board dump on
+   every call by default (`g_Options.fShouldPost`, see `DumpPosition` in
+   gamelist.c), which dominated per-row I/O cost during feature
+   extraction. Fixed by prepending `nopost` to every extraction script.
+8. **Two separate concurrent-process races during interactive mining**,
+   both self-inflicted mid-session: (a) `$!` after a `cd ... && nohup
+   ...` pipeline captured the wrong PID, leaving a stale duplicate
+   process racing on the same output file as a supposedly-fresh rerun;
+   (b) killing that process didn't fully stop it before a new one
+   started. No lasting damage (caught via `ps aux` before trusting any
+   output), but: **always verify a background job is actually singular
+   via `ps aux | grep <script>` before trusting its output**, especially
+   after a kill+relaunch sequence.
+9. **`sizeof(KING_DEFECT_DUMP)` name collision with the
+   `-DKING_DEFECT_DUMP` build flag.** The build flag is defined as
+   literal `1` by the compiler; naming the struct type the same thing
+   caused the preprocessor to mangle the struct declaration wherever the
+   type name appeared. Renamed the struct to `KING_DEFECT_DUMP_INFO`.
+   If you ever add a new `#ifdef SOME_NAME` build flag AND a same-named
+   type/variable in the same session, expect this failure mode.
+10. **`gmake` doesn't recompile on a flag-only change.** After the struct
+    rename (#9) and other `chess.h` edits, a `gmake KING_DEFECT_DUMP=1`
+    without a preceding `gmake clean` silently reused stale `.o` files
+    from an earlier flag-less build, producing a binary that didn't
+    actually have `evalkingdump` registered, with no compile error at
+    all. Always `gmake clean` first when a build-flag combination
+    changes, not just when headers change (this is already called out
+    in `CLAUDE.md` for header changes, but the flag-only case is easy to
+    forget too).
+
+## Performance notes for rerunning
+
+- `pgn/twic_filtered.pgn` (642,268 games) mines in a few minutes when
+  sharded 12-24 ways via `shard_mine.py` (byte-range sharding on real
+  game boundaries, found via a fast raw-text `[Event ` scan). Don't run
+  it unsharded -- a single-process full scan took ~40 minutes.
+- Feature extraction (`extract_king_features.py`) is the slow stage:
+  ~54 minutes for 172,735 rows sharded 18 ways on a contended 24-core
+  box. `nopost` (fix #7 above) is essential -- without it, board-diagram
+  printing dominates I/O cost.
+- `match_controls.py` is pure in-memory bucket lookups, seconds even for
+  the full dataset -- never worth sharding.
+- If `twic_filtered.pgn` (642k games, >=2400 Elo / >40 ply filtered) ever
+  proves insufficient for match density, `pgn/twic.pgn` is 7-8x larger
+  (4,099,094 games, unfiltered) and available as a fallback source --
+  not needed this run (100% match rate achieved on `twic_filtered.pgn`
+  once the above bugs were fixed).
+
+## Suggested next steps
+
+1. Pick 2-3 concrete constant changes from Tier 1/2 above (start with
+   `KingFileDefects`/`KingStormingPawnDefects` -- lowest risk).
+2. Land them as a normal `eval.c` diff, run `precommit_check.sh`.
+3. `sd10` comparison across the three curated suites against
+   `head_reference/`.
+4. `match_play.py` gated head-to-head match against the pre-change DNA.
+5. Only keep the change if `LOWER95 >= 0.5`.
+6. Once a change lands, consider whether `CountKingSafetyDefects` (lazy
+   eval) needs corresponding rework -- deferred by design until a real
+   change is validated (see project goal section above).