Redesign lazy/super-lazy eval margins with position-specific percentile

tables, and fix four real search/recognizer bugs found while validating it

Main theme: replace the flat, heavily-padded LAZY_EVAL_BASE_MARGIN /
SUPER_LAZY_MARGIN_BY_ARMY constants with per-material-bucket percentile
tables (king safety by attacker army, residual by own army, separate
tables for the lazy vs super-lazy baselines) derived from a 30k-position
calibration pool, targeting a measured ~1-1.5% exceeded-margin rate
instead of an unmeasured "add lots of headroom" floor. QSearch's
FUTILITY_BASE_MARGIN_LAZY/SUPERLAZY were re-derived to match the new,
now-meaningful (p50-based) iPositional this produces. Self-play confirms
the working tree beats both HEAD, a more liberal (~5% miss target)
variant, and a more conservative (near-zero miss) variant of the same
margin mechanism.

Along the way, an initial self-play regression (this margin work bundled
with several changes in one pass, one of which broke things) led to a
day of ZFS-snapshot bisection and then targeted DEBUG-build fuzzing
against a stratified 200k-position corpus (tests/cross_sectional_200k.ep_,
new), which is what actually found and fixed four distinct, real,
pre-existing engine bugs -- none of them related to the margin redesign
itself:

- searchsup.c (CommonSearchInit): three early-return paths (timer
  abort, draw, too-deep bailout) left a node's own PV[] row stale
  instead of clearing it, letting an ancestor's UpdatePV splice a
  leftover move from a completely unrelated node into the displayed
  PV -- root cause of a WalkPV/MakeMove assert ("moving a piece to
  capture, but nothing was on the target square").

- dynamic.c (RecordEnprisePieceAtPly): the two-slot en-prise dedup
  compared (square, piece) instead of square alone, so a same-square
  re-record with a different piece (legitimate -- a fail-high
  capture's victim is restored by UnmakeMove and can differ across
  calls at the same ply) shifted a duplicate square into both slots,
  tripping ValueOfMaterialInTroubleDespiteMove's two-distinct-squares
  assert.

- recogn.c (_RecognizeKBKP): the wrong-colored-bishop corner-fortress
  draw shortcut fired even when the weak side has its own pawn(s),
  which can hand the strong side real zugzwang resources the fortress
  logic doesn't account for -- produced a false hard draw bound where
  EGTB shows a near-forced loss.

- eval.c (_EvalPasserRacesAgainstLoneKings): a rook pawn's race was
  credited as winning even when the strong side's own king sits
  exactly on its own pawn's queening square (distance 0 specifically --
  adjacent is fine, the square is empty) -- a classic stalemate
  resource the race heuristic can't see. Only reachable through
  _RecognizeKPK's pawn-hash-cache-hit path, which is why
  testrecogn.c's TestRecognExhaustiveKPK reported "0 wrong" the whole
  time; extended it to cover both color assignments and the full board
  (no mirror-shortcut) and documented the remaining cache-hit coverage
  gap for a future session to close.

Also: match_play.py gains a local_ebf metric (geometric-mean of
depth-normalized node-growth ratios between consecutive completed
iterations) alongside the existing cumulative nodes**(1/depth) ebf,
which turned out to be misleading when comparing engines with
different lazy-exit aggressiveness -- and probe.c/hash.c pick up two
small, independently-found correctness fixes (TB root-probe promotion
piece missing its color bit; ASSERT(g_pHashTable) crashing on the
legitimate "--hash none" config) noticed along the way.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 112 insertions, 0 deletions

diff --git a/src/eval_tune/compare_king_safety_crafty.py b/src/eval_tune/compare_king_safety_crafty.py
new file mode 100644
index 0000000..a018fc7
--- /dev/null
+++ b/src/eval_tune/compare_king_safety_crafty.py
@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+"""Empirical magnitude sanity check: for the same ~500 curated near-mate
+"quiet" positions used to derive the king-safety ring-defect features
+(see mine_king_mates.py / matched_combined_full.csv), compare typhoon's
+own king-safety subscore (iSafety, dumped via KING_DEFECT_DUMP's new
+'realsafety' field -- see chess.h/eval.c/command.c 2026-09-15 additions)
+against Crafty's "kings" row from its `score` command breakdown, for the
+side whose king safety is actually being assessed in each row.
+
+Crafty is spawned fresh (one process per FEN, via `setboard <fen>;
+score; quit`) rather than batched in one long-lived process -- testing
+showed a second setboard+score call inside one Crafty session silently
+reused the FIRST position's cached score, not a real re-evaluation, so
+batching would have silently produced 500 copies of the same number.
+"""
+import csv
+import re
+import subprocess
+import sys
+
+CRAFTY_BIN = "/home/scott/crafty/crafty"
+CRAFTY_CWD = "/home/scott/crafty"
+TYPHOON_KINGDUMP = "/usr/home/scott/typhoon/src/typhoon_kingdump"
+IN_CSV = "/tmp/typhoon/near_mate_500.csv"
+OUT_CSV = "/tmp/typhoon/king_safety_crafty_compare.csv"
+
+KINGS_ROW_RE = re.compile(
+    r"kings\.+\s+([-\d.]+)\s+\|\s+([-\d.]+)\s+([-\d.]+)\s+([-\d.]+)\s+\|\s+"
+    r"([-\d.]+)\s+([-\d.]+)\s+([-\d.]+)\s+\|"
+)
+
+
+def crafty_kings_comp(fen, color):
+    """Returns Crafty's 'kings' row 'comp' column (in pawns) for `color`."""
+    script = f"setboard {fen}\nscore\nquit\n"
+    p = subprocess.run([CRAFTY_BIN], input=script, cwd=CRAFTY_CWD,
+                        capture_output=True, text=True, timeout=30)
+    m = KINGS_ROW_RE.search(p.stdout)
+    if not m:
+        return None
+    white_comp, black_comp = float(m.group(1)), float(m.group(5))
+    return white_comp if color == "white" else black_comp
+
+
+def load_rows():
+    with open(IN_CSV, newline="") as f:
+        return list(csv.DictReader(f))
+
+
+def typhoon_realsafety_batch(rows):
+    """Batches all FENs through one typhoon_kingdump `script` call (this
+    engine's setboard/eval IS known to work correctly across repeated
+    calls in one session -- see extract_king_features.py's run_shard)."""
+    script_path = "/tmp/typhoon/compare_batch.script"
+    out_path = "/tmp/typhoon/compare_batch.out"
+    with open(script_path, "w") as f:
+        f.write("force\nnopost\n")
+        for r in rows:
+            f.write(f"setboard {r['fen']}\n")
+            f.write("evalkingdump\n")
+    cmd = [TYPHOON_KINGDUMP, "--batch", "--command", f"script {script_path}"]
+    with open(out_path, "w") as logf:
+        subprocess.run(cmd, stdout=logf, stderr=subprocess.STDOUT, timeout=300)
+    dumps = []
+    with open(out_path) as f:
+        for line in f:
+            if line.startswith("KINGDEFECTDUMP"):
+                color = re.search(r"color=(\S+)", line).group(1)
+                realsafety = int(re.search(r"realsafety=(-?\d+)", line).group(1))
+                dumps.append((color, realsafety))
+    if len(dumps) != 2 * len(rows):
+        raise RuntimeError(f"expected {2*len(rows)} dump lines, got {len(dumps)}")
+    out = []
+    for i, r in enumerate(rows):
+        black_color, black_rs = dumps[2 * i]
+        white_color, white_rs = dumps[2 * i + 1]
+        assert black_color == "black" and white_color == "white"
+        out.append(black_rs if r["mated_color_or_sampled_color"] == "black" else white_rs)
+    return out
+
+
+def main():
+    rows = load_rows()
+    print(f"Loaded {len(rows)} rows", file=sys.stderr)
+
+    typhoon_vals = typhoon_realsafety_batch(rows)
+    print(f"typhoon: got {len(typhoon_vals)} realsafety values", file=sys.stderr)
+
+    results = []
+    for i, r in enumerate(rows):
+        crafty_comp = crafty_kings_comp(r["fen"], r["mated_color_or_sampled_color"])
+        if crafty_comp is None:
+            continue
+        results.append({
+            "fen": r["fen"],
+            "color": r["mated_color_or_sampled_color"],
+            "typhoon_realsafety_cp": typhoon_vals[i],
+            "crafty_kings_comp_cp": round(crafty_comp * 100, 1),
+        })
+        if (i + 1) % 50 == 0:
+            print(f"  ...{i+1}/{len(rows)} crafty positions done", file=sys.stderr)
+
+    with open(OUT_CSV, "w", newline="") as f:
+        w = csv.DictWriter(f, fieldnames=["fen", "color", "typhoon_realsafety_cp",
+                                           "crafty_kings_comp_cp"])
+        w.writeheader()
+        w.writerows(results)
+    print(f"Wrote {len(results)} rows to {OUT_CSV}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()